Xen devel - Feb 2010 - Memory fragmentation, order>0 allocation, and 4.0 dynamic RAM optimization features

In a recent thread:

http://lists.xensource.com/archives/html/xen-devel/2010-02/msg00295.html

Jan Beulich points out that the memory fragmentation that results
from Transcendent Memory ("tmem") sometimes causes problems for
domain creation and PV migration because the shadow code requires
order=2 allocations and the domain struct is order=4.

Though tmem accelerates fragmentation, I *think* this fragmentation
can occur with page sharing/swapping, and possibly PoD.  In fact,
I think it can occur even with just ballooning.

I think the domain struct issue should be relatively easy to
resolve (though maybe with a large patch), but the shadow code
may be much harder.

But unless the shadow code is also fixed, theoretically 75% of RAM
could be "free" but domain creation/migration failures may occur,
reported only as insufficient memory.

Clearly it''s too late to fix this for 4.0 but, given that 4.0-based
product announcements are likely to emphasize the new 4.0 memory
optimization technologies, it might be good to resolve it very
early in 4.1/xen-unstable development.

Comments?

Are there other known order>0 allocations that might result
in similar issues?

Thanks,
Dan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 08/02/2010 18:13, "Dan Magenheimer"
<dan.magenheimer@oracle.com> wrote:
> In a recent thread:
> 
> http://lists.xensource.com/archives/html/xen-devel/2010-02/msg00295.html
> 
> Jan Beulich points out that the memory fragmentation that results
> from Transcendent Memory ("tmem") sometimes causes problems for
> domain creation and PV migration because the shadow code requires
> order=2 allocations and the domain struct is order=4.
> 
> Though tmem accelerates fragmentation, I *think* this fragmentation
> can occur with page sharing/swapping, and possibly PoD.  In fact,
> I think it can occur even with just ballooning.
> 
> I think the domain struct issue should be relatively easy to
> resolve (though maybe with a large patch), but the shadow code
> may be much harder.
I think everything but the shadow use of order-2 allocations is pretty easy
to fix; just a case of logically carving up the multi-page structures.

I''m not sure, but suspect that fragmenting the shadow allocations may
require extra book-keeping space in the page_info structure, for example.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 19:11 +0000 on 08 Feb (1265656284), Keir Fraser
wrote:
> I think everything but the shadow use of order-2 allocations is pretty easy
> to fix; just a case of logically carving up the multi-page structures.
> 
> I''m not sure, but suspect that fragmenting the shadow allocations
may
> require extra book-keeping space in the page_info structure, for example.
I think it can be done without expanding page_info, but it involves
pretty invasive changes to all the parts of the shadow code that deal
with guest<->shadow mappings and with iterating across shadows.

As I said in response to an earlier email, I think the right thing to do
here is to move pretty much all allocation in Xen to superpage
granularity (and not just because it makes this issue go away!).

Tim

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> Dan Magenheimer <dan.magenheimer@oracle.com> 08.02.10 19:13
>>>
>Are there other known order>0 allocations that might result
>in similar issues?
The passthrough code has data structures which dynamically (nr_irqs-
based) can exceed a page (see pt_irq_create_bind_vtd()).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> Dan Magenheimer <dan.magenheimer@oracle.com> 08.02.10 19:13
>>>
>Are there other known order>0 allocations that might result
>in similar issues?
Interestingly, tmem itself indirectly causes order-1 allocations (through
the use of xmem_pool_create(), sizeof(struct xmem_pool) = 0x18d0
on a non-debug build).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> From: Jan Beulich [mailto:JBeulich@novell.com]
> >>> Dan Magenheimer <dan.magenheimer@oracle.com> 08.02.10
19:13 >>>
> >Are there other known order>0 allocations that might result
> >in similar issues?
> 
> Interestingly, tmem itself indirectly causes order-1 allocations
> (through
> the use of xmem_pool_create(), sizeof(struct xmem_pool) = 0x18d0
> on a non-debug build).
Well that''s embarrassing :-}

But to be fair:
1) tmem is just using xen infrastructure code that suffers from the
   same pervasive problem (though admittedly I added the interface
   to xmalloc_tlsf to enable additional xmem pools to be created)
2) tmem fails semi-gracefully by just turning itself off for
   a domain that fails this order-1 allocation (though it really
   need only disable persistent pools, not all tmem pools)

But ignoring my flimsy excuses, Jan, do you have some debug code
you are using to identify order>0 allocations?  If so, could I
have a copy... and perhaps Keir would consider adding
it post-4.0 to make it easier to search-and-destroy.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> Dan Magenheimer <dan.magenheimer@oracle.com> 18.02.10 17:09
>>>
>But ignoring my flimsy excuses, Jan, do you have some debug code
>you are using to identify order>0 allocations?  If so, could I
>have a copy... and perhaps Keir would consider adding
>it post-4.0 to make it easier to search-and-destroy.
I actually noticed this only as a side effect from a much uglier debugging
patch - observing apparent memory corruption with no apparent pattern
during save/restore/migrate, I finally decided to try a brute force method
and track all allocations. Since you are so eager to point out and fix all
order > 0 allocations, I was quite surprised to see one while tmem
itself initialized its state for Dom0. Hence I thought I''d point it
out. The
patch as it stands is, I think, not really a general debugging aid - if you
think differently, I can of course still share it.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Since you are so eager to point out and fix all order > 0 allocations
"eager" is a rather poor word choice :-)
> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@novell.com]
> Sent: Thursday, February 18, 2010 9:19 AM
> To: Dan Magenheimer
> Cc: Grzegorz Milos; Tim Deegan; PatrickColp; Andrew Peace;
> GeorgeDunlap; Ian Pratt; Keir Fraser; xen-devel@lists.xensource.com
> Subject: RE: Memory fragmentation, order>0 allocation, and 4.0 dynamic
> RAM optimization features
> 
> >>> Dan Magenheimer <dan.magenheimer@oracle.com> 18.02.10
17:09 >>>
> >But ignoring my flimsy excuses, Jan, do you have some debug code
> >you are using to identify order>0 allocations?  If so, could I
> >have a copy... and perhaps Keir would consider adding
> >it post-4.0 to make it easier to search-and-destroy.
> 
> I actually noticed this only as a side effect from a much uglier
> debugging
> patch - observing apparent memory corruption with no apparent pattern
> during save/restore/migrate, I finally decided to try a brute force
> method
> and track all allocations. Since you are so eager to point out and fix
> all
> order > 0 allocations, I was quite surprised to see one while tmem
> itself initialized its state for Dom0. Hence I thought I''d point
it
> out. The
> patch as it stands is, I think, not really a general debugging aid - if
> you
> think differently, I can of course still share it.
> 
> Jan
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hello,

Can I use XenAPI to do remove host from resource pool for XCP?

if there is no XenAPI - can I use xe command to do 


xe pool-eject host-uuid=uuid  ?  on XCP env


Thanks,

Thuymai


      

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

The XenAPI call pool.eject(host) can be used to remove a host from a pool.

http://www.xen.org/files/XenCloud/ocamldoc/apidoc.html?c=pool

"Instruct a pool master to eject a host from the pool"

The xe command should also work.

Cheers,
Dave

From: Chonduy Nguyen [mailto:chonduy9@yahoo.com]
Sent: 18 February 2010 20:47
To: xen-devel@lists.xensource.com
Cc: Dave Scott; Stephen Spector
Subject: xe pool_reject support on XCP


Hello,

Can I use XenAPI to do remove host from resource pool for XCP?

if there is no XenAPI - can I use xe command to do



xe pool-eject host-uuid=uuid  ?  on XCP env





Thanks,



Thuymai









_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

May I ask what kind of event that I can resgiter for XCP? - or any info
regarding to this feature:

 Event Tracking: progress and notification.

can i register event for DOM_CRASH ? 

I looked at the below link:

http://www.xen.org/files/XenCloud/ocamldoc/apidoc.html?c=event

May you point to example code?


Thanks

Thuymai


      

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

Can you point out documentation talking about how XCP Real-time performance
monitoring and alert?

Do you have a process doing this and we can just register alert or you have
XenAPI
to call to get performance info? 

Thanks

ChonDuy/TNguyen


      

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel