Dear all, I installed a xen hypervisor on a server with 96 GB ram. I chose 64Gb ram for one domU called database. However when the domU starts it only has 32GB memory available. I thought the max was up to 1TB. Am I doing something wrong? Operating system: debian amd64 squeeze. Thank you for your time, Bert Some info about my systems: dpkg -l results: ii xen-hypervisor 4.0.1-2 The Xen Hypervisor on AMD64 ii xen-linux-syst 2.6.32+29 Xen system with Linux 2.6 for 64-bit PCs (me ii xen-linux-syst 2.6.32-34squee Xen system with Linux 2.6.32 on 64-bit PCs ( ii xen-qemu-dm-4. 4.0.1-2 Xen Qemu Device Model virtual machine hardwa ii xen-tools 4.2-1 Tools to manage Xen virtual servers ii xen-utils-4.0 4.0.1-2 XEN administrative tools ii xen-utils-comm 4.0.0-1 XEN administrative tools - common files ii xenstore-utils 4.0.1-2 Xenstore utilities for Xen Xm info results: host : xennekeV2 release : 2.6.32-5-xen-amd64 version : #1 SMP Thu May 19 01:16:47 UTC 2011 machine : x86_64 nr_cpus : 24 nr_nodes : 2 cores_per_socket : 6 threads_per_core : 2 cpu_mhz : 2394 hw_caps : bfebfbff:2c100800:00000000:00001f40:029ee3ff:00000000:00000001:00000000 virt_caps : hvm hvm_directio total_memory : 98291 free_memory : 259 node_to_cpu : node0:0,2,4,6,8,10,12,14,16,18,20,22 node1:1,3,5,7,9,11,13,15,17,19,21,23 node_to_memory : node0:0 node1:259 node_to_dma32_mem : node0:0 node1:256 max_node_id : 1 xen_major : 4 xen_minor : 0 xen_extra : .1 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable xen_commandline : placeholder dom0_mem=32G cc_compiler : gcc version 4.4.5 (Debian 4.4.5-10) cc_compile_by : waldi cc_compile_domain : debian.org cc_compile_date : Wed Jan 12 14:04:06 UTC 2011 xend_config_format : 4 Xm list resulst: Name ID Mem VCPUs State Time(s) Domain-0 0 31976 24 r----- 615.0 database 1 65536 1 r----- 9165.6 On the database domU free -m total used free shared buffers cached Mem: 32764 3519 29245 0 11 100 -/+ buffers/cache: 3407 29357 Swap: 2047 4 2043 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Jun 17, 2011 at 5:09 AM, Bert Sarens <Bert.Sarens@ugent.be> wrote:> Dear all, > > I installed a xen hypervisor on a server with 96 GB ram. > I chose 64Gb ram for one domU called database. However when the domU starts > it only has 32GB memory available. > I thought the max was up to 1TB. Am I doing something wrong? > Operating system: debian amd64 squeeze. > > Thank you for your time, > Bert > > Some info about my systems: > > dpkg -l results: > > ii xen-hypervisor 4.0.1-2 The Xen Hypervisor on AMD64 > ii xen-linux-syst 2.6.32+29 Xen system with Linux 2.6 for 64-bit PCs > (me > ii xen-linux-syst 2.6.32-34squee Xen system with Linux 2.6.32 on 64-bit PCs > ( > ii xen-qemu-dm-4. 4.0.1-2 Xen Qemu Device Model virtual machine > hardwa > ii xen-tools 4.2-1 Tools to manage Xen virtual servers > ii xen-utils-4.0 4.0.1-2 XEN administrative tools > ii xen-utils-comm 4.0.0-1 XEN administrative tools - common files > ii xenstore-utils 4.0.1-2 Xenstore utilities for Xen > > Xm info results: > > host : xennekeV2 > release : 2.6.32-5-xen-amd64 > version : #1 SMP Thu May 19 01:16:47 UTC 2011 > machine : x86_64 > nr_cpus : 24 > nr_nodes : 2 > cores_per_socket : 6 > threads_per_core : 2 > cpu_mhz : 2394 > hw_caps : > bfebfbff:2c100800:00000000:00001f40:029ee3ff:00000000:00000001:00000000 > virt_caps : hvm hvm_directio > total_memory : 98291 > free_memory : 259 > node_to_cpu : node0:0,2,4,6,8,10,12,14,16,18,20,22 > node1:1,3,5,7,9,11,13,15,17,19,21,23 > node_to_memory : node0:0 > node1:259 > node_to_dma32_mem : node0:0 > node1:256 > max_node_id : 1 > xen_major : 4 > xen_minor : 0 > xen_extra : .1 > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 > hvm-3.0-x86_32p hvm-3.0-x86_64 > xen_scheduler : credit > xen_pagesize : 4096 > platform_params : virt_start=0xffff800000000000 > xen_changeset : unavailable > xen_commandline : placeholder dom0_mem=32G > cc_compiler : gcc version 4.4.5 (Debian 4.4.5-10) > cc_compile_by : waldi > cc_compile_domain : debian.org > cc_compile_date : Wed Jan 12 14:04:06 UTC 2011 > xend_config_format : 4 > > Xm list resulst: > > Name ID Mem VCPUs State > Time(s) > Domain-0 0 31976 24 r----- > 615.0 > database 1 65536 1 r----- > 9165.6 > > On the database domU > > free -m > total used free shared buffers cached > Mem: 32764 3519 29245 0 11 100 > -/+ buffers/cache: 3407 29357 > Swap: 2047 4 2043 > >How much memory does xm top (xentop) report? -- Todd Deshane http://www.linkedin.com/in/deshantm http://www.xen.org/products/cloudxen.html http://runningxen.com/ _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
xentop - 13:16:37 Xen 4.0.1 2 domains: 1 running, 1 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown Mem: 100650084k total, 100384044k used, 266040k free CPUs: 24 @ 2394MHz NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) MAXMEM(%) VCPU S NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR VBD_RSECT VBD_WSECT SSID database --b--- 17974 0.0 67105024 66.7 67108864 66.7 1 1 281487 153774 2 0 4890212 1998411 40125562 45614704 0 Domain-0 -----r 1378 1.4 31989760 31.8 no limit n/a 2 4 0 0 0 0 0 0 0 0 0 0 Citeren "Todd Deshane" <todd.deshane@xen.org>:> On Fri, Jun 17, 2011 at 5:09 AM, Bert Sarens <Bert.Sarens@ugent.be> wrote: >> Dear all, >> >> I installed a xen hypervisor on a server with 96 GB ram. >> I chose 64Gb ram for one domU called database. However when the domU starts >> it only has 32GB memory available. >> I thought the max was up to 1TB. Am I doing something wrong? >> Operating system: debian amd64 squeeze. >> >> Thank you for your time, >> Bert >> >> Some info about my systems: >> >> dpkg -l results: >> >> ii xen-hypervisor 4.0.1-2 The Xen Hypervisor on AMD64 >> ii xen-linux-syst 2.6.32+29 Xen system with Linux 2.6 for 64-bit PCs >> (me >> ii xen-linux-syst 2.6.32-34squee Xen system with Linux 2.6.32 on 64-bit PCs >> ( >> ii xen-qemu-dm-4. 4.0.1-2 Xen Qemu Device Model virtual machine >> hardwa >> ii xen-tools 4.2-1 Tools to manage Xen virtual servers >> ii xen-utils-4.0 4.0.1-2 XEN administrative tools >> ii xen-utils-comm 4.0.0-1 XEN administrative tools - common files >> ii xenstore-utils 4.0.1-2 Xenstore utilities for Xen >> >> Xm info results: >> >> host : xennekeV2 >> release : 2.6.32-5-xen-amd64 >> version : #1 SMP Thu May 19 01:16:47 UTC 2011 >> machine : x86_64 >> nr_cpus : 24 >> nr_nodes : 2 >> cores_per_socket : 6 >> threads_per_core : 2 >> cpu_mhz : 2394 >> hw_caps : >> bfebfbff:2c100800:00000000:00001f40:029ee3ff:00000000:00000001:00000000 >> virt_caps : hvm hvm_directio >> total_memory : 98291 >> free_memory : 259 >> node_to_cpu : node0:0,2,4,6,8,10,12,14,16,18,20,22 >> node1:1,3,5,7,9,11,13,15,17,19,21,23 >> node_to_memory : node0:0 >> node1:259 >> node_to_dma32_mem : node0:0 >> node1:256 >> max_node_id : 1 >> xen_major : 4 >> xen_minor : 0 >> xen_extra : .1 >> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 >> hvm-3.0-x86_32p hvm-3.0-x86_64 >> xen_scheduler : credit >> xen_pagesize : 4096 >> platform_params : virt_start=0xffff800000000000 >> xen_changeset : unavailable >> xen_commandline : placeholder dom0_mem=32G >> cc_compiler : gcc version 4.4.5 (Debian 4.4.5-10) >> cc_compile_by : waldi >> cc_compile_domain : debian.org >> cc_compile_date : Wed Jan 12 14:04:06 UTC 2011 >> xend_config_format : 4 >> >> Xm list resulst: >> >> Name ID Mem VCPUs State >> Time(s) >> Domain-0 0 31976 24 r----- >> 615.0 >> database 1 65536 1 r----- >> 9165.6 >> >> On the database domU >> >> free -m >> total used free shared buffers cached >> Mem: 32764 3519 29245 0 11 100 >> -/+ buffers/cache: 3407 29357 >> Swap: 2047 4 2043 >> >> > > How much memory does xm top (xentop) report? > > -- > Todd Deshane > http://www.linkedin.com/in/deshantm > http://www.xen.org/products/cloudxen.html > http://runningxen.com/ >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sat, Jun 18, 2011 at 5:17 AM, Bert Sarens <Bert.Sarens@ugent.be> wrote:> xentop - 13:16:37 Xen 4.0.1 > 2 domains: 1 running, 1 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown > Mem: 100650084k totalthis is 95.987400055 GB So Xen sees all of the memory. See Xen best practices for setting dom0 memory http://wiki.xensource.com/xenwiki/XenBestPractices 100384044k used, 266040k free CPUs: 24 @ 2394MHz> NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) MAXMEM(%) > VCPU > S NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR VBD_RSECT > VBD_WSECT > SSID > database --b--- 17974 0.0 67105024 66.7 67108864 66.7 > 1 1 281487 153774 2 0 4890212 1998411 40125562 > 45614704 > 0 > Domain-0 -----r 1378 1.4 31989760 31.8 no limit n/a > 2 > 4 0 0 0 0 0 0 0 0 > 0 > 0 > > > Citeren "Todd Deshane" <todd.deshane@xen.org>: > >> On Fri, Jun 17, 2011 at 5:09 AM, Bert Sarens <Bert.Sarens@ugent.be> wrote: >>> >>> Dear all, >>> >>> I installed a xen hypervisor on a server with 96 GB ram. >>> I chose 64Gb ram for one domU called database. However when the domU >>> starts >>> it only has 32GB memory available. >>> I thought the max was up to 1TB. Am I doing something wrong? >>> Operating system: debian amd64 squeeze. >>> >>> Thank you for your time, >>> Bert >>> >>> Some info about my systems: >>> >>> dpkg -l results: >>> >>> ii xen-hypervisor 4.0.1-2 The Xen Hypervisor on AMD64 >>> ii xen-linux-syst 2.6.32+29 Xen system with Linux 2.6 for 64-bit >>> PCs >>> (me >>> ii xen-linux-syst 2.6.32-34squee Xen system with Linux 2.6.32 on 64-bit >>> PCs >>> ( >>> ii xen-qemu-dm-4. 4.0.1-2 Xen Qemu Device Model virtual machine >>> hardwa >>> ii xen-tools 4.2-1 Tools to manage Xen virtual servers >>> ii xen-utils-4.0 4.0.1-2 XEN administrative tools >>> ii xen-utils-comm 4.0.0-1 XEN administrative tools - common files >>> ii xenstore-utils 4.0.1-2 Xenstore utilities for Xen >>> >>> Xm info results: >>> >>> host : xennekeV2 >>> release : 2.6.32-5-xen-amd64 >>> version : #1 SMP Thu May 19 01:16:47 UTC 2011 >>> machine : x86_64 >>> nr_cpus : 24 >>> nr_nodes : 2 >>> cores_per_socket : 6 >>> threads_per_core : 2 >>> cpu_mhz : 2394 >>> hw_caps : >>> bfebfbff:2c100800:00000000:00001f40:029ee3ff:00000000:00000001:00000000 >>> virt_caps : hvm hvm_directio >>> total_memory : 98291 >>> free_memory : 259 >>> node_to_cpu : node0:0,2,4,6,8,10,12,14,16,18,20,22 >>> node1:1,3,5,7,9,11,13,15,17,19,21,23 >>> node_to_memory : node0:0 >>> node1:259 >>> node_to_dma32_mem : node0:0 >>> node1:256 >>> max_node_id : 1 >>> xen_major : 4 >>> xen_minor : 0 >>> xen_extra : .1 >>> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 >>> hvm-3.0-x86_32p hvm-3.0-x86_64 >>> xen_scheduler : credit >>> xen_pagesize : 4096 >>> platform_params : virt_start=0xffff800000000000 >>> xen_changeset : unavailable >>> xen_commandline : placeholder dom0_mem=32G >>> cc_compiler : gcc version 4.4.5 (Debian 4.4.5-10) >>> cc_compile_by : waldi >>> cc_compile_domain : debian.org >>> cc_compile_date : Wed Jan 12 14:04:06 UTC 2011 >>> xend_config_format : 4 >>> >>> Xm list resulst: >>> >>> Name ID Mem VCPUs State >>> Time(s) >>> Domain-0 0 31976 24 r----- >>> 615.0 >>> database 1 65536 1 r----- >>> 9165.6 >>> >>> On the database domU >>> >>> free -m >>> total used free shared buffers cached >>> Mem: 32764 3519 29245 0 11 100 >>> -/+ buffers/cache: 3407 29357 >>> Swap: 2047 4 2043 >>> >>> >> >> How much memory does xm top (xentop) report? >> >> -- >> Todd Deshane >> http://www.linkedin.com/in/deshantm >> http://www.xen.org/products/cloudxen.html >> http://runningxen.com/ >> > > > >-- Todd Deshane http://www.linkedin.com/in/deshantm http://www.xen.org/products/cloudxen.html http://runningxen.com/ _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Xen indeed sees all the memory. But the client domU does not. On the database domU: free -m total used free shared buffers cached Mem: 32764 3519 29245 0 11 100 -/+ buffers/cache: 3407 29357 Swap: 2047 4 2043 Is this the domUs fault or how can I fix this? Greetings, bert Citeren "Todd Deshane" <todd.deshane@xen.org>:> On Sat, Jun 18, 2011 at 5:17 AM, Bert Sarens <Bert.Sarens@ugent.be> wrote: >> xentop - 13:16:37 Xen 4.0.1 >> 2 domains: 1 running, 1 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown >> Mem: 100650084k total > > this is 95.987400055 GB > > So Xen sees all of the memory. > > See Xen best practices for setting dom0 memory > http://wiki.xensource.com/xenwiki/XenBestPractices > > > > 100384044k used, 266040k free CPUs: 24 @ 2394MHz >> NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) MAXMEM(%) >> VCPU >> S NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR VBD_RSECT >> VBD_WSECT >> SSID >> database --b--- 17974 0.0 67105024 66.7 67108864 66.7 >> 1 1 281487 153774 2 0 4890212 1998411 40125562 >> 45614704 >> 0 >> Domain-0 -----r 1378 1.4 31989760 31.8 no limit n/a >> 2 >> 4 0 0 0 0 0 0 0 0 >> 0 >> 0 >> >> >> Citeren "Todd Deshane" <todd.deshane@xen.org>: >> >>> On Fri, Jun 17, 2011 at 5:09 AM, Bert Sarens <Bert.Sarens@ugent.be> wrote: >>>> >>>> Dear all, >>>> >>>> I installed a xen hypervisor on a server with 96 GB ram. >>>> I chose 64Gb ram for one domU called database. However when the domU >>>> starts >>>> it only has 32GB memory available. >>>> I thought the max was up to 1TB. Am I doing something wrong? >>>> Operating system: debian amd64 squeeze. >>>> >>>> Thank you for your time, >>>> Bert >>>> >>>> Some info about my systems: >>>> >>>> dpkg -l results: >>>> >>>> ii xen-hypervisor 4.0.1-2 The Xen Hypervisor on AMD64 >>>> ii xen-linux-syst 2.6.32+29 Xen system with Linux 2.6 for 64-bit >>>> PCs >>>> (me >>>> ii xen-linux-syst 2.6.32-34squee Xen system with Linux 2.6.32 on 64-bit >>>> PCs >>>> ( >>>> ii xen-qemu-dm-4. 4.0.1-2 Xen Qemu Device Model virtual machine >>>> hardwa >>>> ii xen-tools 4.2-1 Tools to manage Xen virtual servers >>>> ii xen-utils-4.0 4.0.1-2 XEN administrative tools >>>> ii xen-utils-comm 4.0.0-1 XEN administrative tools - common files >>>> ii xenstore-utils 4.0.1-2 Xenstore utilities for Xen >>>> >>>> Xm info results: >>>> >>>> host : xennekeV2 >>>> release : 2.6.32-5-xen-amd64 >>>> version : #1 SMP Thu May 19 01:16:47 UTC 2011 >>>> machine : x86_64 >>>> nr_cpus : 24 >>>> nr_nodes : 2 >>>> cores_per_socket : 6 >>>> threads_per_core : 2 >>>> cpu_mhz : 2394 >>>> hw_caps : >>>> bfebfbff:2c100800:00000000:00001f40:029ee3ff:00000000:00000001:00000000 >>>> virt_caps : hvm hvm_directio >>>> total_memory : 98291 >>>> free_memory : 259 >>>> node_to_cpu : node0:0,2,4,6,8,10,12,14,16,18,20,22 >>>> node1:1,3,5,7,9,11,13,15,17,19,21,23 >>>> node_to_memory : node0:0 >>>> node1:259 >>>> node_to_dma32_mem : node0:0 >>>> node1:256 >>>> max_node_id : 1 >>>> xen_major : 4 >>>> xen_minor : 0 >>>> xen_extra : .1 >>>> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 >>>> hvm-3.0-x86_32p hvm-3.0-x86_64 >>>> xen_scheduler : credit >>>> xen_pagesize : 4096 >>>> platform_params : virt_start=0xffff800000000000 >>>> xen_changeset : unavailable >>>> xen_commandline : placeholder dom0_mem=32G >>>> cc_compiler : gcc version 4.4.5 (Debian 4.4.5-10) >>>> cc_compile_by : waldi >>>> cc_compile_domain : debian.org >>>> cc_compile_date : Wed Jan 12 14:04:06 UTC 2011 >>>> xend_config_format : 4 >>>> >>>> Xm list resulst: >>>> >>>> Name ID Mem VCPUs State >>>> Time(s) >>>> Domain-0 0 31976 24 r----- >>>> 615.0 >>>> database 1 65536 1 r----- >>>> 9165.6 >>>> >>>> On the database domU >>>> >>>> free -m >>>> total used free shared buffers cached >>>> Mem: 32764 3519 29245 0 11 100 >>>> -/+ buffers/cache: 3407 29357 >>>> Swap: 2047 4 2043 >>>> >>>> >>> >>> How much memory does xm top (xentop) report? >>> >>> -- >>> Todd Deshane >>> http://www.linkedin.com/in/deshantm >>> http://www.xen.org/products/cloudxen.html >>> http://runningxen.com/ >>> >> >> >> >> > > > > -- > Todd Deshane > http://www.linkedin.com/in/deshantm > http://www.xen.org/products/cloudxen.html > http://runningxen.com/ >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sat, Jun 18, 2011 at 04:58:13PM +0200, Bert Sarens wrote:> Xen indeed sees all the memory. > Is this the domUs fault or how can I fix this?Yes. The kernel is built to only use a maximum of 32GiB per domain. You can fix this by using a newer kernel, they are not longer restricted memory wise. Bastian -- There is a multi-legged creature crawling on your shoulder. -- Spock, "A Taste of Armageddon", stardate 3193.9 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 6/18/2011 10:29 AM, Bastian Blank wrote:> Yes. The kernel is built to only use a maximum of 32GiB per domain. > > You can fix this by using a newer kernel, they are not longer restricted > memory wise. > > Bastian >Can you elaborate more please? I''ve run into this issue as well, on fairly modern dom0 / domU kernels. Xentop / xm list show 48gb but the domu only shows 32gb. Can you provide any specifics on when/where this was changed? - Nathan _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 7/18/2011 4:03 PM, Nathan March wrote:> > Can you elaborate more please? I''ve run into this issue as well, on > fairly modern dom0 / domU kernels. Xentop / xm list show 48gb but the > domu only shows 32gb. > > Can you provide any specifics on when/where this was changed? > > - NathanFinally found it, if anyone else is looking for it. In the domu kernel: | CONFIG_XEN_MAX_DOMAIN_MEMORY: | | The pseudo-physical to machine address array is sized | according to the maximum possible memory size of a Xen | domain. This array uses 1 page per gigabyte, so there''s no | need to be too stingy here. | | Symbol: XEN_MAX_DOMAIN_MEMORY [=32] | Prompt: Maximum allowed size of a domain in gigabytes | Defined at arch/x86/xen/Kconfig:16 | Depends on: PARAVIRT_GUEST [=y] && XEN [=y] | Location: | -> Processor type and features | -> Paravirtualized guest support (PARAVIRT_GUEST [=y])| | -> Xen guest support (XEN [=y]) - Nathan _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users