Xen devel - Aug 2005 - How to deallocate memory shared between domain and xen

Hi,

  Can someone more experienced with memory management in Xen give me
  some advice?
  I am not sure how should I free memory shared between Xen and a
domain.
  This is part of my effort to port xenoprof to SMP guests.

  I used the code in trace.c as an example of how to allocate
  pages shared between Xen and a domain. This is how the
  shared pages are allocated:
  a) In Xen:
      - allocate pages from Xen heap using
"alloc_xenheap_pages(order)"
      - share each page with domain using "SHARE_PFN_WITH_DOMAIN(page,
domain)"
  b) In kernel
      - reserve contiguous kernel virtual area using "get_vm_area(size,
VM_IOREMAP)"
      - Map shared pages into kernel virtual area using
"direct_remap_area_pages()"

  This seems to be working fine.
  Now, I am not sure what is the sequence of operations to deallocate
these
  shared pages when they are not needed anymore. I spent sometime 
  looking at the code but I still don''t have a clear understanding
  of how memory is managed in Xen. 

  Any help is greatly appreciated, as will save me a lot of time ...


  Thanks

  Renato


     

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 10 Aug 2005, at 02:53, Santos, Jose Renato G wrote:
>
>   This seems to be working fine.
>   Now, I am not sure what is the sequence of operations to deallocate
> these
>   shared pages when they are not needed anymore. I spent sometime
>   looking at the code but I still don''t have a clear understanding
>   of how memory is managed in Xen.
>
>   Any help is greatly appreciated, as will save me a lot of time ...
You want to be able to deallocate them before the domain you were 
sharing with is destroyed? There are no existing examples in Xen of how 
that would be done (things like grant-table pages and shared_info page 
are all destroyed after the domain dies).

Is this the kind of thing you are looking for?:
  1. On allocaction: SHARE_PFN_WITH_DOMAIN(); get_page();
  2. On deallocation: put_page()

...and then the pages get freed up automatically when the guest has 
freed all references. That is: the pages do not get freed until you 
have given up your one reference (put_page) and the guest has given up 
its references.

This can be added quite easily, but will require an new page flag.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> >   This seems to be working fine.
> >   Now, I am not sure what is the sequence of operations to 
> deallocate 
> > these
> >   shared pages when they are not needed anymore. I spent sometime
> >   looking at the code but I still don''t have a clear
understanding
> >   of how memory is managed in Xen.
> >
> >   Any help is greatly appreciated, as will save me a lot of time ...
> 
> You want to be able to deallocate them before the domain you 
> were sharing with is destroyed? There are no existing 
> examples in Xen of how that would be done (things like 
> grant-table pages and shared_info page are all destroyed 
> after the domain dies).
> 
> Is this the kind of thing you are looking for?:
>   1. On allocaction: SHARE_PFN_WITH_DOMAIN(); get_page();
>   2. On deallocation: put_page()
> 
> ...and then the pages get freed up automatically when the 
> guest has freed all references. That is: the pages do not get 
> freed until you have given up your one reference (put_page) 
> and the guest has given up its references.
> 
> This can be added quite easily, but will require an new page flag.
We''re talking about a pretty special situation here, and I''m
not totally
sure its worth adding this to Xen, at least it requires disucssion.

Jose: as an interim measure, please can you knock up a version that just
deallocates when the domain dies. It would be very useful to get SMP
xen-oprofile working ASAP!

I would like to point out that I currently don''t like the way oprofile
offers EIP samples to each domain for conversion into symbols. I''d much
rather see this done off-line. In my view, Xen should log the
domain,EIP,guest_pagetable,and kernel stack pointer in each sample,
storing it in a per-(physical)CPU buffer that''s shared with the
monitoring domain (usually dom0). Dom0 will then be able to distinguish
different processes (by pagetable) and even different threads (by linux
kernel stack pointer) even if it won''t actually by able to name them.

Comments? (Any how, don''t let my views stop you from getting something
working quickly)

Ian

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ian, Keir

  Thanks for the feedback. 
  Comments embedded in the message
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Ian Pratt
> Sent: Wednesday, August 10, 2005 2:44 AM
> To: Keir Fraser; Santos, Jose Renato G
> Cc: Jose Renato Santos; xen-devel@lists.xensource.com
> Subject: RE: [Xen-devel] How to deallocate memory shared 
> between domain and xen
> 
> 
> > >   This seems to be working fine.
> > >   Now, I am not sure what is the sequence of operations to
> > deallocate
> > > these
> > >   shared pages when they are not needed anymore. I spent sometime
> > >   looking at the code but I still don''t have a clear
understanding
> > >   of how memory is managed in Xen.
> > >
> > >   Any help is greatly appreciated, as will save me a lot 
> of time ...
> > 
> > You want to be able to deallocate them before the domain you
> > were sharing with is destroyed? There are no existing 
> > examples in Xen of how that would be done (things like 
> > grant-table pages and shared_info page are all destroyed 
> > after the domain dies).
> > 
> > Is this the kind of thing you are looking for?:
> >   1. On allocaction: SHARE_PFN_WITH_DOMAIN(); get_page();
> >   2. On deallocation: put_page()
> > 
> > ...and then the pages get freed up automatically when the
> > guest has freed all references. That is: the pages do not get 
> > freed until you have given up your one reference (put_page) 
> > and the guest has given up its references.
> > 
> > This can be added quite easily, but will require an new page flag.
> 
> We''re talking about a pretty special situation here, and
I''m
> not totally sure its worth adding this to Xen, at least it 
> requires disucssion.
> 
> Jose: as an interim measure, please can you knock up a 
> version that just deallocates when the domain dies. It would 
> be very useful to get SMP xen-oprofile working ASAP!
> 
  OK. For now I will not explictly de-allocate the shared memory,
  and let the memory be deallocated when the domain dies.
  I am assuming nothing special has to be done and the memory gets 
  deallocated automaticaly when the domain dies 
  (SHARE_PFN_WITH_DOMAIN() seems to be including the shared 
  page in the list of pages owned by the domain which
  should be automatically de-alocated when the domain dies, right?).
  Will this also cause the memory to be returned to the Xen heap?

  I am assuming this is a temporary solution. 
  The problem is that the memory is mapped into kernel space
  by oprofile which can be configured as a kernel module. 
  Therefore one would expect that the memory would be deallocated
  when the module is unloaded. Assuming we cannot deallocate
  memory explictly we need to remember the first allocation,
  to avoid memory leaks after successive module load/unload.
  Since the module can be unloaded and loaded again this
  info need to saved somewhere in the kernel but outside 
  oprofile module.(argh!, this is ugly ...)

  I just finished most of the logic for SMP guest support
  in Xenoprof, and I am starting to test/debug now. I suspect
  it will take several days to fix all the bugs. 
  I am seeing a real nasty bug right now.
  The machine auto reboots at random times when running xenoprof
  with a SMP dom0 (no error log or message on the console). 
  This is not a deterministic bug and sometimes I can run
  oprofile for a long time (although I just get samples
  for one of the the 2 cpus). I will switch to testing
  on domU and see if the machine survives the bug in this case
  BTW, Do you have any suggestion on what is the best way to debug
  this type of bug in Xen?
   
   
> I would like to point out that I currently don''t like the way 
> oprofile offers EIP samples to each domain for conversion 
> into symbols. I''d much rather see this done off-line. In my 
> view, Xen should log the domain,EIP,guest_pagetable,and 
> kernel stack pointer in each sample, storing it in a 
> per-(physical)CPU buffer that''s shared with the monitoring 
> domain (usually dom0). Dom0 will then be able to distinguish 
> different processes (by pagetable) and even different threads 
> (by linux kernel stack pointer) even if it won''t actually by 
> able to name them.
> 
  I am not sure how we would decode EIP samples into 
  binary images with this approach. Being able to distinguish
  process and threads do not solve this mapping problem. 
  Oprofile maps EIP into binary images by looking at which files
  are mapped in which regions of the process address space and
  then converts the EIP into a file identifier (cookie) and an offset.
  Not sure how we would do this with your suggestion. Also, keep in
  mind that we want to re-use most of existing oprofile code, instead
  of creating a new profiling tool.

  Thanks

  Renato

> Comments? (Any how, don''t let my views stop you from getting 
> something working quickly)
> 
> Ian
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 11 Aug 2005, at 04:41, Santos, Jose Renato G wrote:
>  OK. For now I will not explictly de-allocate the shared memory,
>   and let the memory be deallocated when the domain dies.
>   I am assuming nothing special has to be done and the memory gets
>   deallocated automaticaly when the domain dies
>   (SHARE_PFN_WITH_DOMAIN() seems to be including the shared
>   page in the list of pages owned by the domain which
>   should be automatically de-alocated when the domain dies, right?).
>   Will this also cause the memory to be returned to the Xen heap?
No, you need to deallocate yourself. Add a hook function off 
domain_destruct(), and do any necessary deallocation there. The refcnts 
are to make sure the domain isn;t killed while other domains may have 
references to its xen-heap pages.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel