hi, I would like to share some pages between my domU graphics frontend device, and the backend which runs in userspace in dom0. Right now I am doing this with my own scheme, but presumably grant tables would be the correct solution. Is it possible to use grant tables from dom0 userspace? There used to be a file called tools/libxc/xc_gnttab.c but that seems to be gone now. Jacob _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hey Grant tables aren''t the right interface for use from userspace, which is why tools/libxc/xc_gnttab.c was removed. You should probably just write a kernel module to do the mapping for you using grant tables and then expose a custom interface to userspace from the module to trigger the mapping as you need. Christopher On 3/9/06, Jacob Gorm Hansen <jacobg@diku.dk> wrote:> hi, > > I would like to share some pages between my domU graphics frontend > device, and the backend which runs in userspace in dom0. Right now I > am doing this with my own scheme, but presumably grant tables would be > the correct solution. > > Is it possible to use grant tables from dom0 userspace? There used to > be a file called tools/libxc/xc_gnttab.c but that seems to be gone > now. > > Jacob > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 9 Mar 2006, at 17:59, Jacob Gorm Hansen wrote:> I would like to share some pages between my domU graphics frontend > device, and the backend which runs in userspace in dom0. Right now I > am doing this with my own scheme, but presumably grant tables would be > the correct solution. > > Is it possible to use grant tables from dom0 userspace? There used to > be a file called tools/libxc/xc_gnttab.c but that seems to be gone > now.I think blktap gives an example how to do this, but it might be specific to aio right now. You could probably use some of the same hooks to provide a device file that you could mmap(), passing grant refs to map. I cc''ed Andy Warfield in case he has any ideas... The only other supported mechanism is the xc_foreign mapping interfaces. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
(Combining w/ Christopher''s reply:) I would add my support to providing this as a common service to userspace (as opposed to just creating a solution that works for your project). Using xc_foreign_mapping interfaces may work for dom0 code but it requires the domain to be privileged and does not have fine-grained permissions (i.e. if you can map something then you can map anything). Grant tables are a much better, from a security POV, solution to sharing memory. Joseph Cihula (Linux) Software Security Architect Open Source Technology Center Intel Corp. *** These opinions are not necessarily those of my employer *** On Thursday, March 09, 2006 10:13 AM, Keir Fraser <> wrote:> I think blktap gives an example how to do this, but it might be > specific to aio right now. You could probably use some of the same > hooks to provide a device file that you could mmap(), passing grant > refs to map. I cc''ed Andy Warfield in case he has any ideas... > > The only other supported mechanism is the xc_foreign mapping > interfaces.On Thursday, March 09, 2006 10:10 AM, Christopher Clark <> wrote:> Grant tables aren''t the right interface for use from userspace, which > is why tools/libxc/xc_gnttab.c was removed. You should probably just > write a kernel module to do the mapping for you using grant tables and > then expose a custom interface to userspace from the module to trigger > the mapping as you need.> On 9 Mar 2006, at 17:59, Jacob Gorm Hansen wrote: > >> I would like to share some pages between my domU graphics frontend >> device, and the backend which runs in userspace in dom0. Right now I >> am doing this with my own scheme, but presumably grant tables would >> be the correct solution. >> >> Is it possible to use grant tables from dom0 userspace? There used to >> be a file called tools/libxc/xc_gnttab.c but that seems to be gone >> now._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I think blktap gives an example how to do this, but it might be > specific to aio right now. You could probably use some of the same > hooks to provide a device file that you could mmap(), passing grant > refs to map. I cc''ed Andy Warfield in case he has any ideas...The blktap code in unstable is a bit out of date at the moment, but should have sufficient examples to do what you want. Because of AIO, the code in there arranges for parallel user and kernel mappings, and then annotates the vm area so that get_user_pages can find the appropriate page_struct when AIO requests are decoded down into BIOs. If you just want to mmap some memory from another domain into user through a char device, then you can trim a lot of that code out. In linux/drivers/xen/blktap/blktap.c: dispatch_rw_block_io() -- decodes a batch of block requests, and maps the grants to kernel and user. As I say, you should be able to leave out all the kernel mappings and vm area updates. blktap_read_ufe_ring() -- does the associated unmappings. There''s a bunch of code at the top to set up the device node and manage the mmap area. The code is a bit dated, but maybe it''s helpful... a. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I agree user-mode mappings would be a very useful addition to Xen. *** However *** Following much of Andrew''s work in my own driver, I''ve tried to create general purpose user-mode mappings based on grant tables. The results are unsatisfactory. You''ll encounter some tricky domain crashes that have been discussed already on this list. I''ve hacked around many of the problems, such as implicit unmapping of grant references, but others remain. Some of the issues have no resolution in the grant table architecture. If you have something working now, I''d stick with it. IMHO, before user-mode mappings can be robustly supported, we''ll have to leave grant tables for a fresh approach. -steve -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cihula, Joseph Sent: Thursday, March 09, 2006 10:25 AM To: Keir Fraser; Jacob Gorm Hansen Cc: Andrew Warfield; xen-devel Devel Subject: RE: [Xen-devel] Grant tables from dom0 userspace? (Combining w/ Christopher''s reply:) I would add my support to providing this as a common service to userspace (as opposed to just creating a solution that works for your project). Using xc_foreign_mapping interfaces may work for dom0 code but it requires the domain to be privileged and does not have fine-grained permissions (i.e. if you can map something then you can map anything). Grant tables are a much better, from a security POV, solution to sharing memory. Joseph Cihula (Linux) Software Security Architect Open Source Technology Center Intel Corp. *** These opinions are not necessarily those of my employer *** On Thursday, March 09, 2006 10:13 AM, Keir Fraser <> wrote:> I think blktap gives an example how to do this, but it might be > specific to aio right now. You could probably use some of the same > hooks to provide a device file that you could mmap(), passing grant > refs to map. I cc''ed Andy Warfield in case he has any ideas... > > The only other supported mechanism is the xc_foreign mapping > interfaces.On Thursday, March 09, 2006 10:10 AM, Christopher Clark <> wrote:> Grant tables aren''t the right interface for use from userspace, which > is why tools/libxc/xc_gnttab.c was removed. You should probably just > write a kernel module to do the mapping for you using grant tables and> then expose a custom interface to userspace from the module to trigger> the mapping as you need.> On 9 Mar 2006, at 17:59, Jacob Gorm Hansen wrote: > >> I would like to share some pages between my domU graphics frontend >> device, and the backend which runs in userspace in dom0. Right now I >> am doing this with my own scheme, but presumably grant tables would >> be the correct solution. >> >> Is it possible to use grant tables from dom0 userspace? There used to>> be a file called tools/libxc/xc_gnttab.c but that seems to be gone >> now._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 9 Mar 2006, at 18:46, King, Steven R wrote:> Following much of Andrew''s work in my own driver, I''ve tried to create > general purpose user-mode mappings based on grant tables. The results > are unsatisfactory. You''ll encounter some tricky domain crashes that > have been discussed already on this list.With due respect, just because you haven''t got it working correctly yet does not mean it can''t be done. It''s working okay in the blktap driver after all. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Steven,> I''ve hacked around many of > the problems, such as implicit unmapping of grant references, but others > remain. Some of the issues have no resolution in the grant table > architecture.It would be quite useful to know what specific problems you have encountered in this effort as insight for future versions of the code. The main problem that I have encountered in user-level foreign page mappings is (as discussed previously) that Linux is a bit hasty in blowing away user virtual memory areas, and doesn''t provide a good mechanism to safely unmap the pages. The kernel has all the information it needs to do the right thing though, and I think this should be reasonably fixed in the future -- i don''t see a compelling reason for Xen to do extra book-keeping to cover for something that the OS could just as easily do. Especially for something like cross-VM mappings, which don''t exist in the non-virtualized system. Are there implicit unmapping problems that you thing would remain unsolved if, for instance, there was a vm area destructor early enough to allow the OS to properly unmap active grants? When you say, "others remain," can you provide a little more detail? thanks! a. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thursday 09 March 2006 11:59, Jacob Gorm Hansen wrote:> > I would like to share some pages between my domU graphics frontend > device, and the backend which runs in userspace in dom0. Right now I > am doing this with my own scheme, but presumably grant tables would be > the correct solution. > > Is it possible to use grant tables from dom0 userspace? There used to > be a file called tools/libxc/xc_gnttab.c but that seems to be gone > now.Rusty did a userspace block driver you may also want to look at. He was publishing bundles here: http://ozlabs.org/~rusty/xen/bundles/ See also his mail on the subject, titled "[BUNDLE] Testing a simpler inter-domain transport" and later "[BUNDLE] Xen Share w/ block device". -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Fair enough. Not being an expert, I''m willing to take my lumps when voicing opinion on these issues. No offense to Andrew, who is very helpful, but I don''t believe blktap has worked around these grant table problems: Shared pages can only be unshared if all the mapping domains play nice -- IMHO, an "Enterprise-Grade" security problem already discussed here: http://lists.xensource.com/archives/html/xen-devel/2006-01/msg00369.html In the code, this manifests itself as a "WARNING: g.e. still in use" printk in sparse/arch/xen/kernel/gnttab.c The implicit grant unmap problem, previously discussed here: http://lists.xensource.com/archives/html/xen-devel/2006-02/msg00517.html http://lists.xensource.com/archives/html/xen-devel/2006-01/msg00689.html This is an interesting one, since it''s arguable where "paravirtualization" ends and "unnatural" begins. It would sure be nice to have a scheme that didn''t force the gamut of guest OS''s to add special hooks just for Xen shared pages. Whether Xen adds bookkeeping, or all guest OS''s create a special case, I''m in favor of whatever scheme has the least net complexity. The reliance on the _PAGE_GNTMAP bit in the pte to catch disallowed OS behavior, such as implicit unmaps above. I recall the code comment saying using pte bits is broken for *BSD for example. If implicit unmaps can be made to work, then perhaps this bit goes away. If only explicit unmaps are allowed (via new OS hooks or whatever), then I think we''re be stuck with _PAGE_GNTMAP. -steve -----Original Message----- From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] Sent: Thursday, March 09, 2006 10:50 AM To: King, Steven R Cc: Andrew Warfield; Cihula, Joseph; Jacob Gorm Hansen; xen-devel Devel Subject: Re: [Xen-devel] Grant tables from dom0 userspace? On 9 Mar 2006, at 18:46, King, Steven R wrote:> Following much of Andrew''s work in my own driver, I''ve tried to create> general purpose user-mode mappings based on grant tables. The results> are unsatisfactory. You''ll encounter some tricky domain crashes that > have been discussed already on this list.With due respect, just because you haven''t got it working correctly yet does not mean it can''t be done. It''s working okay in the blktap driver after all. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Andrew --> Are there implicit unmapping problems that you thing would remainunsolved if,> for instance, there was a vm area destructor early enough to allow theOS to> properly unmap active grants?I''m having a post-unmap balloon driver problem, but could be my fault (see below). I''m not convinced that implicit unmapping is an unreasonable thing to ask of Xen. It feels less complex overall to have Xen deal with this once, rather than N operating system deal with it N different ways. I''d be interested to hear arguments here.> When you say, "others remain," can you provide a little more detail?I tried to list some in my reply to Keir. While I''ve got you on the line... :^) I''m having two problems right now with my attempt at implicit unmap. The hacks let me limp along well enough to make progress on the code that uses the shared memory, but any thoughts on further improvements are most welcome. The implicit unmap code executes when the cleanup_writable_pagetable() code _PAGE_GNTMAP flag is found in a modified pte. The backtrace looks like this: (XEN) [<ff13603e>] put_page_from_l1e+0xd0/0x1af (XEN) [<ff13a891>] revalidate_l1+0x159/0x168 (XEN) [<ff13aac1>] ptwr_flush+0x221/0x32f (XEN) [<ff13b6a7>] cleanup_writable_pagetable+0x5c/0x7d (XEN) [<ff137c20>] do_mmuext_op+0x85/0x8c1 (XEN) [<ff149e0f>] hypercall+0x8f/0xaf At this point, the Linux has just squashed the pte. Xen code knows the l1e, and I''ve added a few more bits to the maptrack object to allow me to find a match from the corresponding pfn. It''s kludgey, because this only works in the special case where only one map track entity exists for a given pfn per domain. This is problem #1. Ideas? If we had the exact address of the pte, there would be no ambiguity in which maptrack entry owns the mapping. I nosed around and did not find a way to get the pte addr, but still hold out hope that it''s possible. Armed with the "correct" maptrack entry, I can do most of the ummapping work then and there. Later, I get the "late" vma_close() from the OS where my guest driver explicitly unmaps to finish the job. All is well, and I''m back at the guest command prompt. Problem #2 is that the balloon driver crashes me if I try to restart my user-level app. The dump looks like this: kernel BUG at drivers/xen/balloon/balloon.c:218! invalid operand: 0000 [#1] SMP Modules linked in: CPU: 0 EIP: 0061:[<c023d8c4>] Not tainted VLI EFLAGS: 00010097 (2.6.14-xenU) EIP is at increase_reservation+0x1c4/0x230 eax: c1000000 ebx: 0000f377 ecx: c11e6eec edx: c0042000 esi: 00000004 edi: c11e6ee0 ebp: c038bf18 esp: c038bed8 ds: 007b es: 007b ss: 0069 Process events/0 (pid: 5, threadinfo=c038a000 task=c03b3530) Stack: c11e6e80 c0346d00 00000000 00000000 cf3ac000 cf3ac000 00000004 00000000 00000000 00007ff0 cf8f5030 c03b3658 00000005 00000004 00000000 c038a000 c038bf34 c023db8a 00000004 c0132dff 00000000 c03b1c00 c02ff004 c038bfb4 Call Trace: [<c010845f>] show_stack+0x7f/0xa0 [<c0108612>] show_registers+0x162/0x1d0 [<c0108824>] die+0xf4/0x180 [<c02adb85>] do_trap+0xb5/0xc0 [<c0108bec>] do_invalid_op+0xbc/0xd0 [<c0108127>] error_code+0x2b/0x30 [<c023db8a>] balloon_process+0x3a/0xb0 [<c012e05c>] worker_thread+0x19c/0x240 [<c0132ada>] kthread+0xba/0xc0 [<c0105f49>] kernel_thread_helper+0x5/0xc Code: 00 00 00 00 31 d2 89 f8 e8 5a 51 f0 ff ff 45 cc 8b 55 08 39 55 cc 0f 82 6b ff ff ff e9 2b ff ff ff 0f 0b e7 00 3a 92 2d c0 eb cd <0f> 0b da 00 3a 92 2d c0 e9 76 ff ff ff 0f 0b d7 00 3a 92 2d c0 I haven''t pursed this one at all, so it could be the result a dumb mistake on my part. -steve -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Andrew Warfield Sent: Thursday, March 09, 2006 10:59 AM To: King, Steven R Cc: Cihula, Joseph; Jacob Gorm Hansen; xen-devel Devel Subject: Re: [Xen-devel] Grant tables from dom0 userspace? Hi Steven,> I''ve hacked around many of > the problems, such as implicit unmapping of grant references, but > others remain. Some of the issues have no resolution in the grant > table architecture.It would be quite useful to know what specific problems you have encountered in this effort as insight for future versions of the code. The main problem that I have encountered in user-level foreign page mappings is (as discussed previously) that Linux is a bit hasty in blowing away user virtual memory areas, and doesn''t provide a good mechanism to safely unmap the pages. The kernel has all the information it needs to do the right thing though, and I think this should be reasonably fixed in the future -- i don''t see a compelling reason for Xen to do extra book-keeping to cover for something that the OS could just as easily do. Especially for something like cross-VM mappings, which don''t exist in the non-virtualized system. Are there implicit unmapping problems that you thing would remain unsolved if, for instance, there was a vm area destructor early enough to allow the OS to properly unmap active grants? When you say, "others remain," can you provide a little more detail? thanks! a. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> At this point, the Linux has just squashed the pte. Xen code knows the > l1e, and I''ve added a few more bits to the maptrack object to allow me > to find a match from the corresponding pfn. It''s kludgey, because this > only works in the special case where only one map track entity exists > for a given pfn per domain. This is problem #1. Ideas? If we had the > exact address of the pte, there would be no ambiguity in which maptrack > entry owns the mapping. I nosed around and did not find a way to get > the pte addr, but still hold out hope that it''s possible. > > Armed with the "correct" maptrack entry, I can do most of the ummapping > work then and there. Later, I get the "late" vma_close() from the OS > where my guest driver explicitly unmaps to finish the job.If you are going down the implicit unmap path, I don''t think that trying to sleuth out the PTE after the fact and based on partial information is the right way to go. I''d think it would make more sense to store the PTE that the grant has been bound to explicity and hook the implicit unmap off of the pte update validation code in Xen. I''d be interested to hear what keir thinks though... iirc, the original concern with this sort of approach was the space overhead of storing all the outstanding mapping -- the maptracks are intentionally very lightweight.> All is well, and I''m back at the guest command prompt. Problem #2 is > that the balloon driver crashes me if I try to restart my user-level > app. The dump looks like this:I don''t see a BUG() at line 218 of my version of balloon.c, but presumably you are failing one of the sanity checks in increase_reservation. Sounds like there may be a bug hunting trip in your future. ;) a. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel