Xen devel - Jul 2013 - Shared memory between a process in dom0 and a process in domU

Hello,

I''m trying to share memory pages between a process executing in dom0
and a process executing in a domU. I work with xen 4.3 and linux 3.2
(it''s necessary due to a compatibility issue). I succeeded in accessing
pages from a process in dom0, but I''m encountering problems to access
them from a domU. I tried two methods :
1) I allocate a vm area in which I put the pages identified by their grant
references. This step works, but I don''t find how to retrieve the pages
in the mmap function to map them into user space. Here is the corresponding code
(it''s based on the xensocket code) :

if (!(x->buffer_area = alloc_vm_area(buffer_num_pages * PAGE_SIZE, NULL))) {
    DPRINTK("error: cannot allocate %d buffer pages\n",
buffer_num_pages);
    goto err_unmap;
}

//x->buffer_addr is a pointer used to access more easily to
x->buffer_area->addr
x->buffer_addr = (unsigned long)x->buffer_area->addr;

//we retrieve the first gref
grefp = &d->buffer_first_gref;
for (i = 0; i < buffer_num_pages; i++) {

    memset(&op, 0, sizeof(op));
    op.host_addr = x->buffer_addr + i * PAGE_SIZE;
    op.flags = GNTMAP_host_map;
    op.ref = *grefp;
    op.dom = x->otherend_id;

    rc = HYPERVISOR_grant_table_op(GNTTABOP_map_grant_ref, &op, 1);
    if (rc == -ENOSYS) {
        goto err_unmap;
    }

    if (op.status) {
           DPRINTK("error: grant table mapping failed\n");
        goto err_unmap;
    }

    x->buffer_handles[i] = op.handle;
    grefp = (int *)(x->buffer_addr + i * PAGE_SIZE);
}

Now, the grefs are mapped and the data they contain is accessible from the
module. I instantiated a mmap function to access them from a user process :

//I first tried to get the PFN of the page to map it with remap_pfn_range, but
it doesn''t work as vmalloc_to_pfn returns 0 and not the PFN
unsigned long pfn = vmalloc_to_pfn((void
*)(x->buffer_addr+recv_offset*PAGE_SIZE));
if(pfn_valid(pfn)){
    printk(KERN_INFO "PFN is valide\n");
}

//I studied how gntdev worked and I realised it uses vm_insert_page, I then
tried to retrieve the struct page instead of the PFN
struct page * shPage = vmalloc_to_page((void
*)(x->buffer_addr+recv_offset*PAGE_SIZE));
if(shPage == NULL)
    printk(KERN_INFO "Page not found\n");

vma->vm_flags |= VM_RESERVED;

//This function fails with the error code -22 (invalid value)
int err = vm_insert_page(vma, vma->vm_start, shPage);
if (err)
    printk(KERN_INFO "Error while inserting a page\n");
else
    printk(KERN_INFO "Page inserted !!");

I don''t understand why this method fails. vmalloc_to_page returns a
struct page *, but the function page_count (called by vm_insert_page) fails when
called for this page. Is there a subtlety specific to Xen to allow this
operation to succeed ?

2) As the first method didn''t succeed, I used what was done in gntdev
(using pages instead of vm area to "mount" granted references) :

if((x->pages = kcalloc(buffer_num_pages, sizeof(x->pages[0]),
GFP_KERNEL))==NULL){
    printk(KERN_INFO "No space left on device\n");
    goto err;
}

if(alloc_xenballooned_pages(buffer_num_pages, x->pages, false)){
    printk(KERN_INFO "No place anymore for ballooned pages");
    goto err;
}

//I did the test for 1 page only
for (i = 0; i < 1; i++) {
    memset(&op, 0, sizeof(op));
    //I succeed here to get the PFN and the kaddr corresponding with the page
    unsigned long pfn = page_to_pfn(x->pages[i]);
    gnttab_set_map_op(&op, (unsigned long) pfn_to_kaddr(pfn),
GNTMAP_host_map, *grefp, x->otherend_id);
    rc = gnttab_map_refs(&op,NULL,x->pages+i, 1);
    if(rc){
        printk(KERN_INFO "Error while mapping ref\n");
        goto err_unmap;
    }
    else if (op.status) {
        DPRINTK("error: grant table mapping failed\n");
        goto err_unmap;
    }
    else
        printk(KERN_INFO "Page mapped\n");
}

I don''t really know what happens here, but when gnttab_map_refs is
called, the calling domU is rebooted and its ID is changed (from 1 to 2 i.e.).

Can someone please tell me what''s wrong in my code ? I just have to get
the right page in the first method to make it work.

Thank you.

Best regards,

Sebastien Fremal


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Wed, 2013-07-24 at 14:46 +0000, Sébastien FREMAL [530784]
wrote:
> Hello,
> 
> I'm trying to share memory pages between a process executing in dom0
> and a process executing in a domU. I work with xen 4.3 and linux 3.2
> (it's necessary due to a compatibility issue). I succeeded in
> accessing pages from a process in dom0, but I'm encountering problems
> to access them from a domU. I tried two methods :
> 1) I allocate a vm area in which I put the pages identified by their
> grant references. This step works, but I don't find how to retrieve
> the pages in the mmap function to map them into user space. Here is
> the corresponding code (it's based on the xensocket code) :
> 
> if (!(x->buffer_area = alloc_vm_area(buffer_num_pages * PAGE_SIZE,
> NULL))) {
>     DPRINTK("error: cannot allocate %d buffer pages\n",
> buffer_num_pages);
>     goto err_unmap;
> }
> 
> //x->buffer_addr is a pointer used to access more easily to
> x->buffer_area->addr
> x->buffer_addr = (unsigned long)x->buffer_area->addr;
> 
> //we retrieve the first gref
> grefp = &d->buffer_first_gref;
> for (i = 0; i < buffer_num_pages; i++) {
> 
>     memset(&op, 0, sizeof(op));
>     op.host_addr = x->buffer_addr + i * PAGE_SIZE;
>     op.flags = GNTMAP_host_map;
>     op.ref = *grefp;
>     op.dom = x->otherend_id;
> 
>     rc = HYPERVISOR_grant_table_op(GNTTABOP_map_grant_ref, &op, 1);
>     if (rc == -ENOSYS) {
>         goto err_unmap;
>     }
> 
>     if (op.status) {
>            DPRINTK("error: grant table mapping failed\n");
>         goto err_unmap;
>     }
> 
>     x->buffer_handles[i] = op.handle;
>     grefp = (int *)(x->buffer_addr + i * PAGE_SIZE);
> }
> 
> Now, the grefs are mapped and the data they contain is accessible from
> the module. I instantiated a mmap function to access them from a user
> process :
> 
> //I first tried to get the PFN of the page to map it with
> remap_pfn_range, but it doesn't work as vmalloc_to_pfn returns 0 and
> not the PFN
> unsigned long pfn = vmalloc_to_pfn((void *)(x->buffer_addr
> +recv_offset*PAGE_SIZE));
> if(pfn_valid(pfn)){
>     printk(KERN_INFO "PFN is valide\n");
> }
> 
> //I studied how gntdev worked and I realised it uses vm_insert_page, I
> then tried to retrieve the struct page instead of the PFN
> struct page * shPage = vmalloc_to_page((void *)(x->buffer_addr
> +recv_offset*PAGE_SIZE));
> if(shPage == NULL)
>     printk(KERN_INFO "Page not found\n");
> 
> vma->vm_flags |= VM_RESERVED;
> 
> //This function fails with the error code -22 (invalid value)
> int err = vm_insert_page(vma, vma->vm_start, shPage);
> if (err)
>     printk(KERN_INFO "Error while inserting a page\n");
> else
>     printk(KERN_INFO "Page inserted !!");
> 
> I don't understand why this method fails. vmalloc_to_page returns a
> struct page *, but the function page_count (called by vm_insert_page)
> fails when called for this page.
What do you mean by "fails"? That function returns the current value
of
the page's reference count it cannot really fail as such.
>  Is there a subtlety specific to Xen to allow this operation to
> succeed ?
Not as far as I know, your issue appears to be a generic Linux one. I
don't know the answer off hand but you might want to try one of the
Linux lists/newgroups perhaps?
> I don't really know what happens here, but when gnttab_map_refs is
> called, the calling domU is rebooted and its ID is changed (from 1 to
> 2 i.e.).
Perhaps the guest tried something so illegal that it was killed and the
toolstack rebooted it? You should see evidence of this in either the
hypervisor logs or the toolstack logs.
> Can someone please tell me what's wrong in my code ? I just have to
> get the right page in the first method to make it work.
If what you want to do is share memory between dom0 and domU userspace
then you should look into the gntdev and gntshr device drivers which
already exist to support this usecase and will mean you don't need to
write any drivers at all.

For an example of the use of these existing drivers you could look at
libvchan (tools/libvchan). In fact if what you want is a shared memory
ring protocol between two processes then libvchan may already do
everything you need.

Ian.




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Hello,

Thank you for your answer. As the struct page * I get with vmalloc_to_page
is not a valid reference, page_count returns 0 and the function
vm_insert_page returns with an error :

int vm_insert_page(struct vm_area_struct *vma, unsigned long addr,
                         struct page *page)
{
         if (addr < vma->vm_start || addr >= vma->vm_end)
                 return -EFAULT;
         if (!page_count(page))
                 return -EINVAL;
         if (!(vma->vm_flags & VM_MIXEDMAP)) {
                 BUG_ON(down_read_trylock(&vma->vm_mm->mmap_sem));
                 BUG_ON(vma->vm_flags & VM_PFNMAP);
                 vma->vm_flags |= VM_MIXEDMAP;
         }
         return insert_page(vma, addr, page, vma->vm_page_prot);
}

I already studied the xen drivers and libvchan to understand how xen
mechanisms work, but they don''t operate exactly the way I wish (I
search to
transfer data with a ring buffer without any copy). I decided to work in
the user space instead of the kernel space as I have less problem there. I
will use xen-evtchn and xen-gntdev to use kernel features.

Best regards,

Sebastien Fremal


2013/7/29 Ian Campbell <Ian.Campbell@citrix.com>
> On Wed, 2013-07-24 at 14:46 +0000, Sébastien FREMAL [530784] wrote:
> > Hello,
> >
> > I''m trying to share memory pages between a process executing
in dom0
> > and a process executing in a domU. I work with xen 4.3 and linux 3.2
> > (it''s necessary due to a compatibility issue). I succeeded in
> > accessing pages from a process in dom0, but I''m encountering
problems
> > to access them from a domU. I tried two methods :
> > 1) I allocate a vm area in which I put the pages identified by their
> > grant references. This step works, but I don''t find how to
retrieve
> > the pages in the mmap function to map them into user space. Here is
> > the corresponding code (it''s based on the xensocket code) :
> >
> > if (!(x->buffer_area = alloc_vm_area(buffer_num_pages * PAGE_SIZE,
> > NULL))) {
> >     DPRINTK("error: cannot allocate %d buffer pages\n",
> > buffer_num_pages);
> >     goto err_unmap;
> > }
> >
> > //x->buffer_addr is a pointer used to access more easily to
> > x->buffer_area->addr
> > x->buffer_addr = (unsigned long)x->buffer_area->addr;
> >
> > //we retrieve the first gref
> > grefp = &d->buffer_first_gref;
> > for (i = 0; i < buffer_num_pages; i++) {
> >
> >     memset(&op, 0, sizeof(op));
> >     op.host_addr = x->buffer_addr + i * PAGE_SIZE;
> >     op.flags = GNTMAP_host_map;
> >     op.ref = *grefp;
> >     op.dom = x->otherend_id;
> >
> >     rc = HYPERVISOR_grant_table_op(GNTTABOP_map_grant_ref, &op,
1);
> >     if (rc == -ENOSYS) {
> >         goto err_unmap;
> >     }
> >
> >     if (op.status) {
> >            DPRINTK("error: grant table mapping failed\n");
> >         goto err_unmap;
> >     }
> >
> >     x->buffer_handles[i] = op.handle;
> >     grefp = (int *)(x->buffer_addr + i * PAGE_SIZE);
> > }
> >
> > Now, the grefs are mapped and the data they contain is accessible from
> > the module. I instantiated a mmap function to access them from a user
> > process :
> >
> > //I first tried to get the PFN of the page to map it with
> > remap_pfn_range, but it doesn''t work as vmalloc_to_pfn
returns 0 and
> > not the PFN
> > unsigned long pfn = vmalloc_to_pfn((void *)(x->buffer_addr
> > +recv_offset*PAGE_SIZE));
> > if(pfn_valid(pfn)){
> >     printk(KERN_INFO "PFN is valide\n");
> > }
> >
> > //I studied how gntdev worked and I realised it uses vm_insert_page, I
> > then tried to retrieve the struct page instead of the PFN
> > struct page * shPage = vmalloc_to_page((void *)(x->buffer_addr
> > +recv_offset*PAGE_SIZE));
> > if(shPage == NULL)
> >     printk(KERN_INFO "Page not found\n");
> >
> > vma->vm_flags |= VM_RESERVED;
> >
> > //This function fails with the error code -22 (invalid value)
> > int err = vm_insert_page(vma, vma->vm_start, shPage);
> > if (err)
> >     printk(KERN_INFO "Error while inserting a page\n");
> > else
> >     printk(KERN_INFO "Page inserted !!");
> >
> > I don''t understand why this method fails. vmalloc_to_page
returns a
> > struct page *, but the function page_count (called by vm_insert_page)
> > fails when called for this page.
>
> What do you mean by "fails"? That function returns the current
value of
> the page''s reference count it cannot really fail as such.
>
> >  Is there a subtlety specific to Xen to allow this operation to
> > succeed ?
>
> Not as far as I know, your issue appears to be a generic Linux one. I
> don''t know the answer off hand but you might want to try one of
the
> Linux lists/newgroups perhaps?
>
> > I don''t really know what happens here, but when
gnttab_map_refs is
> > called, the calling domU is rebooted and its ID is changed (from 1 to
> > 2 i.e.).
>
> Perhaps the guest tried something so illegal that it was killed and the
> toolstack rebooted it? You should see evidence of this in either the
> hypervisor logs or the toolstack logs.
>
> > Can someone please tell me what''s wrong in my code ? I just
have to
> > get the right page in the first method to make it work.
>
> If what you want to do is share memory between dom0 and domU userspace
> then you should look into the gntdev and gntshr device drivers which
> already exist to support this usecase and will mean you don''t need
to
> write any drivers at all.
>
> For an example of the use of these existing drivers you could look at
> libvchan (tools/libvchan). In fact if what you want is a shared memory
> ring protocol between two processes then libvchan may already do
> everything you need.
>
> Ian.
>
>
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel