Xen devel - Feb 2008 - Sharing Memory between userspace of dom0 and userspace of domU

I''m trying to better understand how xen shares memory between domains.
Ultimately what I would like to do is share memory between a userspace
tool in dom0 and a userspace application running in domU. Is that
currently possible?

The two methods for sharing memory that I found were grant tables and
using xc_map_foreign_range(). Grant tables seem to be just for kernel
space? I found xc_gnttab_map_grant_ref and other xc_gnttab_ userspace
API, but that requires a grant reference and I don''t see a way to
generate or even get a grant reference from userspace.

Second, I tried xc_map_foreign_range() from my dom0 userspace program.
I started up a userspace program in domU and told it to malloc several
bytes of memory and print out the virtual address returned by malloc.
It wrote the string "hello" to the memory location and then just sat
in a loop. Then I started up a userspace program in dom0 and ran
xc_translate_foreign_address on the given virtual address and used
that as my MFN for xc_map_foreign_range. No errors were returned but
the memory mapped to dom0 is just garbage compared to "hello". When I
specify an arbitrary virtual address for xc_translate_foreign_address,
it generally gives me an error saying that address is not valid. So I
assume I''m doing something right, but I must be missing something.
What am I doing wrong with this?

If none of these methods are supposed to be used in this manner, is
there any way to share memory between user programs in different
domains like I''m trying to do?

Thanks,
Kareem Dana

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Kareem,

On Feb 17, 2008 11:54 PM, Kareem Dana <kareem.dana@gmail.com>
wrote:
> I''m trying to better understand how xen shares memory between
domains.
> Ultimately what I would like to do is share memory between a userspace
> tool in dom0 and a userspace application running in domU. Is that
> currently possible?
At present, this isn''t supported in the tools, however there is no
technological reason why we couldn''t do this.
> The two methods for sharing memory that I found were grant tables and
> using xc_map_foreign_range(). Grant tables seem to be just for kernel
> space? I found xc_gnttab_map_grant_ref and other xc_gnttab_ userspace
> API, but that requires a grant reference and I don''t see a way to
> generate or even get a grant reference from userspace.
At the moment, yes, the only way to grant access to a page is from the
kernel. This is due to the fact that kernel memory corresponds to
physical memory, and we don''t have to worry about interactions with
the swapper, or what happens when the process dies.
> Second, I tried xc_map_foreign_range() from my dom0 userspace program.
> I started up a userspace program in domU and told it to malloc several
> bytes of memory and print out the virtual address returned by malloc.
> It wrote the string "hello" to the memory location and then just
sat
> in a loop. Then I started up a userspace program in dom0 and ran
> xc_translate_foreign_address on the given virtual address and used
> that as my MFN for xc_map_foreign_range. No errors were returned but
> the memory mapped to dom0 is just garbage compared to "hello".
When I
> specify an arbitrary virtual address for xc_translate_foreign_address,
> it generally gives me an error saying that address is not valid. So I
> assume I''m doing something right, but I must be missing something.
> What am I doing wrong with this?
One possibility is that your process is not active when you run
xc_translate_foreign_address: this function works on the basis of a
VCPU id, and your process must have its page directory base address in
CR3 on that VCPU for the translation to succeed.
> If none of these methods are supposed to be used in this manner, is
> there any way to share memory between user programs in different
> domains like I''m trying to do?
You''re going to need a kernel module to achieve this. I can think of
two approaches:

* Allocate a buffer in the kernel, grant another domain access to this
buffer, and mmap it into your process. (I''ve managed to get this to
work, but it''s not yet robust.)
* Allocate a buffer in userspace, then use get_user_pages and kmap to
map it into the kernel, and then grant access to the other domain.
(Michael Abd-El-Malek had success with this method, as described in
this thread:
http://lists.xensource.com/archives/html/xen-devel/2008-01/msg01204.html
)

Hope this helps!

Regards,

Derek Murray.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks Derek. This info helped a lot. I''ll try to post my code/results
back when I get some more time to work on a kernel module.


On Feb 18, 2008 4:58 AM, Derek Murray <Derek.Murray@cl.cam.ac.uk>
wrote:
> Hi Kareem,
>
> On Feb 17, 2008 11:54 PM, Kareem Dana <kareem.dana@gmail.com> wrote:
> > I''m trying to better understand how xen shares memory between
domains.
> > Ultimately what I would like to do is share memory between a userspace
> > tool in dom0 and a userspace application running in domU. Is that
> > currently possible?
>
> At present, this isn''t supported in the tools, however there is no
> technological reason why we couldn''t do this.
>
> > The two methods for sharing memory that I found were grant tables and
> > using xc_map_foreign_range(). Grant tables seem to be just for kernel
> > space? I found xc_gnttab_map_grant_ref and other xc_gnttab_ userspace
> > API, but that requires a grant reference and I don''t see a
way to
> > generate or even get a grant reference from userspace.
>
> At the moment, yes, the only way to grant access to a page is from the
> kernel. This is due to the fact that kernel memory corresponds to
> physical memory, and we don''t have to worry about interactions
with
> the swapper, or what happens when the process dies.
>
> > Second, I tried xc_map_foreign_range() from my dom0 userspace program.
> > I started up a userspace program in domU and told it to malloc several
> > bytes of memory and print out the virtual address returned by malloc.
> > It wrote the string "hello" to the memory location and then
just sat
> > in a loop. Then I started up a userspace program in dom0 and ran
> > xc_translate_foreign_address on the given virtual address and used
> > that as my MFN for xc_map_foreign_range. No errors were returned but
> > the memory mapped to dom0 is just garbage compared to
"hello". When I
> > specify an arbitrary virtual address for xc_translate_foreign_address,
> > it generally gives me an error saying that address is not valid. So I
> > assume I''m doing something right, but I must be missing
something.
> > What am I doing wrong with this?
>
> One possibility is that your process is not active when you run
> xc_translate_foreign_address: this function works on the basis of a
> VCPU id, and your process must have its page directory base address in
> CR3 on that VCPU for the translation to succeed.
>
> > If none of these methods are supposed to be used in this manner, is
> > there any way to share memory between user programs in different
> > domains like I''m trying to do?
>
> You''re going to need a kernel module to achieve this. I can think
of
> two approaches:
>
> * Allocate a buffer in the kernel, grant another domain access to this
> buffer, and mmap it into your process. (I''ve managed to get this
to
> work, but it''s not yet robust.)
> * Allocate a buffer in userspace, then use get_user_pages and kmap to
> map it into the kernel, and then grant access to the other domain.
> (Michael Abd-El-Malek had success with this method, as described in
> this thread:
http://lists.xensource.com/archives/html/xen-devel/2008-01/msg01204.html
> )
>
> Hope this helps!
>
> Regards,
>
> Derek Murray.
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi --
> > At the moment, yes, the only way to grant access to a page is from the
> > kernel. This is due to the fact that kernel memory corresponds to
> > physical memory, and we don''t have to worry about
interactions with
> > the swapper, or what happens when the process dies.
>From what I understand of what you''ve said, are you saying that the
shared memory pages that granted access must always be in physical
memory and cannot be swapped out, even if the guest kernel decided to
for some reason?  Does Xen enforce this in any way, e.g. pinning the
pages somehow?

Thanks,
Mike

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi --
> > At the moment, yes, the only way to grant access to a page is from the
> > kernel. This is due to the fact that kernel memory corresponds to
> > physical memory, and we don''t have to worry about
interactions with
> > the swapper, or what happens when the process dies.
>From what I understand of what you''ve said, are you saying that the
shared memory pages that granted access must always be in physical
memory and cannot be swapped out, even if the guest kernel decided to
for some reason?  Does Xen enforce this in any way, e.g. pinning the
pages somehow?

Thanks,
Mike

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Mike,

On Feb 19, 2008 4:16 PM, Mike Sun <msun@gatech.edu>
wrote:
> > > At the moment, yes, the only way to grant access to a page is
from the
> > > kernel. This is due to the fact that kernel memory corresponds to
> > > physical memory, and we don''t have to worry about
interactions with
> > > the swapper, or what happens when the process dies.
>
> From what I understand of what you''ve said, are you saying that
the
> shared memory pages that granted access must always be in physical
> memory and cannot be swapped out, even if the guest kernel decided to
> for some reason?  Does Xen enforce this in any way, e.g. pinning the
> pages somehow?
A shared (granted) page is shared based on its (G)MFN. There is in
fact no interaction with Xen when granting a page, as this can be done
by simply writing to the grant table.

The granted physical page is pinned when it is mapped, however, this
only means that, if the granting domain dies, the page is not freed
erroneously.

However, as far as I can tell, the granting domain is free to do
whatever it likes with the physical page. Therefore, if the process
containing the granted page dies, you need to keep a reference to the
physical page that was granted, because another domain has mapped it
and can therefore read the contents of the page, or overwrite them.
This could cause a security problem or unexpected behaviour in the
granting domain.

Likewise, if the kernel decided to swap out the page that you granted,
and replace it with another virtual page, you would not  observe the
effect of granting access to a particular virtual address (which is
all you would know about in user-space). Therefore you would have to
pin the page using mlock() or something similar.

I hope this makes things clearer, but let me know if anything I''ve
said doesn''t make sense.

Regards,

Derek Murray.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, 2008-02-19 at 11:16 -0500, Mike Sun wrote:
> Hi --
> 
> > > At the moment, yes, the only way to grant access to a page is
from the
> > > kernel. This is due to the fact that kernel memory corresponds to
> > > physical memory, and we don''t have to worry about
interactions with
> > > the swapper, or what happens when the process dies.
> 
> >From what I understand of what you''ve said, are you saying
that the
> shared memory pages that granted access must always be in physical
> memory and cannot be swapped out, even if the guest kernel decided to
> for some reason?  Does Xen enforce this in any way, e.g. pinning the
> pages somehow?
Xen never swaps out domain pages. So there''s no need to pin them on the
side of the VMM. I believe that will answer your original question
regarding swapping out of domain page tables. 

Regarding swapping by domains: For memory mapped across domains, or
shared with the VMM by dom0 *tools*, the control library will presently
take care to properly mlock() the pages. Kernel space memory (e.g.
shared by driver pairs) is typically not swappable, or will get marked
accordingly. Generally, it''s the domain''s own responsibility
to keep the
maps consistent, as it is solely its own security and stability which
suffers otherwise.

Regarding you question how Xen distinguishes page frame types (i.e. page
tables): A page frame becomes a page table to Xen, as soon as the domain
makes it so (e.g. by linking cr3 to it, or referencing it in the page
directory,... etc.). The overall mechanism is that all (most) of these
operations are done with hypercalls. There''s no writable access to page
tables, and Xen enforces this. Xen will declare the memory a PT and keep
a reference count which keeps it that way. Take a look into the Xen
interface manual and/or e.g. arch/x86/mm.c if you''re interested in
that.

regards,
Daniel

-- 
Daniel Stodden
LRR     -      Lehrstuhl für Rechnertechnik und Rechnerorganisation
Institut für Informatik der TU München             D-85748 Garching
http://www.lrr.in.tum.de/~stodden         mailto:stodden@cs.tum.edu
PGP Fingerprint: F5A4 1575 4C56 E26A 0B33  3D80 457E 82AE B0D8 735B



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks Daniel, Derek.  I think that cleared up a good number of things for me.

Regarding page table pages...  you mentioned:
> Xen never swaps out domain pages. So there''s no need to pin them
on the
> side of the VMM. I believe that will answer your original question
> regarding swapping out of domain page tables.
My question is what would happen if the guest OS swaps out page table
pages and the mfn backing that PT page is now allocated to a normal
user page.  How would Xen know that the type of that page has changed
if this were allowed to happen?

Thanks,
Mike

On Feb 19, 2008 12:02 PM, Daniel Stodden <stodden@cs.tum.edu>
wrote:
>
>
> On Tue, 2008-02-19 at 11:16 -0500, Mike Sun wrote:
> > Hi --
> >
> > > > At the moment, yes, the only way to grant access to a page
is from the
> > > > kernel. This is due to the fact that kernel memory
corresponds to
> > > > physical memory, and we don''t have to worry about
interactions with
> > > > the swapper, or what happens when the process dies.
> >
> > >From what I understand of what you''ve said, are you
saying that the
> > shared memory pages that granted access must always be in physical
> > memory and cannot be swapped out, even if the guest kernel decided to
> > for some reason?  Does Xen enforce this in any way, e.g. pinning the
> > pages somehow?
>
> Xen never swaps out domain pages. So there''s no need to pin them
on the
> side of the VMM. I believe that will answer your original question
> regarding swapping out of domain page tables.
>
> Regarding swapping by domains: For memory mapped across domains, or
> shared with the VMM by dom0 *tools*, the control library will presently
> take care to properly mlock() the pages. Kernel space memory (e.g.
> shared by driver pairs) is typically not swappable, or will get marked
> accordingly. Generally, it''s the domain''s own
responsibility to keep the
> maps consistent, as it is solely its own security and stability which
> suffers otherwise.
>
> Regarding you question how Xen distinguishes page frame types (i.e. page
> tables): A page frame becomes a page table to Xen, as soon as the domain
> makes it so (e.g. by linking cr3 to it, or referencing it in the page
> directory,... etc.). The overall mechanism is that all (most) of these
> operations are done with hypercalls. There''s no writable access to
page
> tables, and Xen enforces this. Xen will declare the memory a PT and keep
> a reference count which keeps it that way. Take a look into the Xen
> interface manual and/or e.g. arch/x86/mm.c if you''re interested in
that.
>
> regards,
> Daniel
>
> --
> Daniel Stodden
> LRR     -      Lehrstuhl für Rechnertechnik und Rechnerorganisation
> Institut für Informatik der TU München             D-85748 Garching
> http://www.lrr.in.tum.de/~stodden         mailto:stodden@cs.tum.edu
> PGP Fingerprint: F5A4 1575 4C56 E26A 0B33  3D80 457E 82AE B0D8 735B
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

h
On Tue, 2008-02-19 at 13:34 -0500, Mike Sun wrote:
> Thanks Daniel, Derek.  I think that cleared up a good number of things for
me.
> 
> Regarding page table pages...  you mentioned:
> > Xen never swaps out domain pages. So there''s no need to pin
them on the
> > side of the VMM. I believe that will answer your original question
> > regarding swapping out of domain page tables.
> 
> My question is what would happen if the guest OS swaps out page table
> pages and the mfn backing that PT page is now allocated to a normal
> user page.  How would Xen know that the type of that page has changed
> if this were allowed to happen?
The page type refcount drops to zero when the page isn''t linked into
the
page table structure anymore (see the page_info struct definition in the
Xen source tree for types and refcounts). So the guest just unlinks the
PT frame (via a hypercall). From there on, the guest may reuse the frame
under a writable mapping. Note that this is consistent with overall OS
behavior: First the PT usage is cleared, then the frame reused,
independent of whether the contents are swapped to disk, the process
died, etc.

Vice versa for writable (RW) types: If the guest has a writable mapping
on that page (type=RW, count>0) somewhere, attempts to use it as a PT
frame elsewhere will fail.

regards,
daniel

-- 
Daniel Stodden
LRR     -      Lehrstuhl für Rechnertechnik und Rechnerorganisation
Institut für Informatik der TU München             D-85748 Garching
http://www.lrr.in.tum.de/~stodden         mailto:stodden@cs.tum.edu
PGP Fingerprint: F5A4 1575 4C56 E26A 0B33  3D80 457E 82AE B0D8 735B



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks Daniel,
> The page type refcount drops to zero when the page isn''t linked
into the
> page table structure anymore (see the page_info struct definition in the
> Xen source tree for types and refcounts). So the guest just unlinks the
> PT frame (via a hypercall). From there on, the guest may reuse the frame
> under a writable mapping. Note that this is consistent with overall OS
> behavior: First the PT usage is cleared, then the frame reused,
> independent of whether the contents are swapped to disk, the process
> died, etc.
So it is possible for the guest OS to swap out a page table page from
what you''re saying?  It seems like that would be a problem for the
domain save/migration mechanism since in that process PT pages are
canonicalized from virtual-to-machine mappings to
virtual-to-pseudophysical mappings...  if any PT pages are swapped to
disk and not canonicalized, they would have incorrect mappings.
Someone has suggested that Xen is aware of any swaps made by the guest
OS (by way of hypercalls to update the PTEs) and canonicalizes the
page table pages then.  Are you aware if that''s the case?  I still
have to go through that code to find out.

Thanks,
Mike

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel