Frank S Fejes III
2010-Mar-18 13:28 UTC
[Xen-users] Xen Hypervisor memory utilization (leak?)
Hello, everybody. Over the past couple months I''ve noticed that a few
of our Xen servers are reporting (via xentop) hypervisor memory
utilization discrepancies between the sum of VM memory allocations and
the actual memory used. In time, the free memory drops to a point
where no new VMs can be started. Below is sample xentop output from
yesterday with extra columns chopped. The hypervisor reports that it
is using 32gb of ram, however the sum of the running VMs is 16gb.
Ballooning is disabled and dom0_mem is set to 4gb.
xentop - 15:38:20 Xen 3.4.1_19718_04-2.1
12 domains: 2 running, 10 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
Mem: 33549248k total, 33241552k used, 307696k free CPUs: 4 @ 2992MHz
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) MAXMEM(%)
xxxxxxxxxx --b--- 1584 0.2 547840 1.6 547840 1.6
Domain-0 -----r 165894 34.3 4192768 12.5 no limit n/a
xxxxxxxxxx --b--- 15166 22.3 3071916 9.2 3076096 9.2
xxxxxxxxxx -----r 1328 77.8 1028012 3.1 1028096 3.1
xxxxxxxxxx --b--- 55433 4.7 1027880 3.1 1028096 3.1
xxxxxxxxxx --b--- 67527 4.8 1027880 3.1 1028096 3.1
xxxxxxxxxx --b--- 73019 6.8 1027880 3.1 1028096 3.1
xxxxxxxxxx --b--- 61266 6.8 1027880 3.1 1028096 3.1
xxxxxxxxxx --b--- 78880 6.2 1028012 3.1 1028096 3.1
xxxxxxxxxx --b--- 969 7.7 516012 1.5 516096 1.5
xxxxxxxxxx --b--- 10544 1.5 1027880 3.1 1028096 3.1
xxxxxxxxxx --b--- 21001 5.8 1027880 3.1 1028096 3.1
Our dom0 is opensuse 11.2 and, as you can see above, our Xen version
is 3.4.1 which is provided by the opensuse project. This behavior is
only seen on our "development" Xen servers where we routinely
create/clone/destroy a large number of (mostly HVM) VMs, however when
doing any of those functions I cannot reproduce the "leak". That is
to say, when I create a VM the hypervisor memory free statistic drops
in an amount corresponding to the size of the VM. When the VM is
destroyed, the free statistic increases back to where it was
previously.
Has anyone else seen this behavior before? I have not been able to
find any way to get the hypervisor to reclaim some of the lost memory
and, as a result, I have been forced to reboot the server. On systems
with 20+ VMs where we don''t have the option for live migration this
can be rather traumatic.
Any help or insight would be greatly appreciated!
--frank
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-Mar-18 19:27 UTC
Re: [Xen-users] Xen Hypervisor memory utilization (leak?)
On Thu, Mar 18, 2010 at 08:28:05AM -0500, Frank S Fejes III wrote:> Hello, everybody. Over the past couple months I''ve noticed that a few > of our Xen servers are reporting (via xentop) hypervisor memory > utilization discrepancies between the sum of VM memory allocations and > the actual memory used. In time, the free memory drops to a point > where no new VMs can be started. Below is sample xentop output from > yesterday with extra columns chopped. The hypervisor reports that it > is using 32gb of ram, however the sum of the running VMs is 16gb. > Ballooning is disabled and dom0_mem is set to 4gb. > > xentop - 15:38:20 Xen 3.4.1_19718_04-2.1 > 12 domains: 2 running, 10 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown > Mem: 33549248k total, 33241552k used, 307696k free CPUs: 4 @ 2992MHz > NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) MAXMEM(%) > xxxxxxxxxx --b--- 1584 0.2 547840 1.6 547840 1.6 > Domain-0 -----r 165894 34.3 4192768 12.5 no limit n/a > xxxxxxxxxx --b--- 15166 22.3 3071916 9.2 3076096 9.2 > xxxxxxxxxx -----r 1328 77.8 1028012 3.1 1028096 3.1 > xxxxxxxxxx --b--- 55433 4.7 1027880 3.1 1028096 3.1 > xxxxxxxxxx --b--- 67527 4.8 1027880 3.1 1028096 3.1 > xxxxxxxxxx --b--- 73019 6.8 1027880 3.1 1028096 3.1 > xxxxxxxxxx --b--- 61266 6.8 1027880 3.1 1028096 3.1 > xxxxxxxxxx --b--- 78880 6.2 1028012 3.1 1028096 3.1 > xxxxxxxxxx --b--- 969 7.7 516012 1.5 516096 1.5 > xxxxxxxxxx --b--- 10544 1.5 1027880 3.1 1028096 3.1 > xxxxxxxxxx --b--- 21001 5.8 1027880 3.1 1028096 3.1 > > Our dom0 is opensuse 11.2 and, as you can see above, our Xen version > is 3.4.1 which is provided by the opensuse project. This behavior is > only seen on our "development" Xen servers where we routinely > create/clone/destroy a large number of (mostly HVM) VMs, however when > doing any of those functions I cannot reproduce the "leak". That is > to say, when I create a VM the hypervisor memory free statistic drops > in an amount corresponding to the size of the VM. When the VM is > destroyed, the free statistic increases back to where it was > previously. > > Has anyone else seen this behavior before? I have not been able to > find any way to get the hypervisor to reclaim some of the lost memory > and, as a result, I have been forced to reboot the server. On systems > with 20+ VMs where we don''t have the option for live migration this > can be rather traumatic. > > Any help or insight would be greatly appreciated! >I think you should report this to Novell bugzilla! -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users