Mark & All, Short version: If a domU driver provides an application with a mapping to a shared page, where in the driver should the corresponding GNTTABOP_unmap_grant_ref call reside to avoid a domain crash? Long version: I''m trying fix my code running in a domU to avoid Xen calling domain_crash() in mm.c line 615. This occurs when a domain "implicitly unmaps a granted PTE". The somewhat abridged version of events: 1) Remote domU shares a page 2) app in local domU calls mmap() 3) driver mmap code calls GNTTAB_map_grant_ref() and returns the mapping to the app 4) app accesses the shared page without problem 5) app is done, calls munmap() 6) Xen kills the domain in mm.c line 615. I have not yet found a way to make the GNTTABOP_unmap_grant_ref() call in time to avoid the crash. The driver release() function is too late and there''s no hook in struct file_operations to handle unmap explicitly. Mark Williamson''s blktap driver is calling GNTTAB_unmap_grant_ref, but it''s unclear to me if the same unmap()''ing problem exists there too. Any help is much appreciated. -steve _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Jan 24, 2006 at 03:28:55PM -0800, King, Steven R wrote:> I have not yet found a way to make the GNTTABOP_unmap_grant_ref() call > in time to avoid the crash. The driver release() function is too late > and there''s no hook in struct file_operations to handle unmap > explicitly.Could you hook the vma->close function? ISTR it gets called on unmap. If this is too fine grained or is not called on every munmap (unmapping of partial VMAs?), you could clean up all mappings when the file that the application mmap''s is closed. Cheers, Muli -- Muli Ben-Yehuda http://www.mulix.org | http://mulix.livejournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Vma->close() didn''t work. Thanks for the great suggestion though. :^) To verify that I wasn''t making some new mistake, inspection of linux/mm/mmap.c shows that Linux calls vma->close() *after* the page table manipulation. In do_munmap(), the sequence is: detach_vmas_to_be_unmapped() unmap_region() <--- page table manipulation buried in here remove_vma_list() <--- vma->close() buried in here Is this turning into a Xen problem? Why does Xen forbid implicit unmapping of shared pages? -steve -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Muli Ben-Yehuda Sent: Tuesday, January 24, 2006 3:50 PM To: King, Steven R Cc: xen-devel; Mark Williamson Subject: Re: [Xen-devel] GNTTABOP_unmap_grant_ref On Tue, Jan 24, 2006 at 03:28:55PM -0800, King, Steven R wrote:> I have not yet found a way to make the GNTTABOP_unmap_grant_ref() call> in time to avoid the crash. The driver release() function is too late> and there''s no hook in struct file_operations to handle unmap > explicitly.Could you hook the vma->close function? ISTR it gets called on unmap. If this is too fine grained or is not called on every munmap (unmapping of partial VMAs?), you could clean up all mappings when the file that the application mmap''s is closed. Cheers, Muli -- Muli Ben-Yehuda http://www.mulix.org | http://mulix.livejournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I''m trying fix my code running in a domU to avoid Xen calling > domain_crash() in mm.c line 615. This occurs when a domain "implicitly > unmaps a granted PTE". The somewhat abridged version of events: > > 1) Remote domU shares a page > 2) app in local domU calls mmap() > 3) driver mmap code calls GNTTAB_map_grant_ref() and returns the mapping > to the app > 4) app accesses the shared page without problem > 5) app is done, calls munmap() > 6) Xen kills the domain in mm.c line 615.Linux doesn''t understand that the memory you''ve mmap-ed is "magic" and it''s trying to simply zap the PTEs that refer to it. Because those were installed using the grant tables mechanisms, it looks like a domain bug to Xen - you can''t mix managing PTEs using grant tables *and* writable pagetables (side note: I''d like to remove this restriction at some point, since XenFS is going to run up against it). I''d expect that this *might* be a problem if the system starts swapping... if Linux thinks it can fool around with your PTEs, it almost certainly thinks it can do other undesirable stuff. We need to keep the memory management routines well away from granted memory in this case!> I have not yet found a way to make the GNTTABOP_unmap_grant_ref() call > in time to avoid the crash. The driver release() function is too late > and there''s no hook in struct file_operations to handle unmap > explicitlyI agree with Muli - it''s a property of the VMA, not of the file, etc.It''s unfortunate hooking the destructor didn''t work - but all is not lost. You need to to take away control of that memory from the core memory management algorithms. In the olden days you might have tried setting the RESERVED bit for the struct page(s) in question. This is no longer the Done Thing - it''s a bit more complex... See: http://lwn.net/Articles/160501/ (for background info from the recent past) http://lwn.net/Articles/162860/ (for the current state of affairs) You''ll need a 2.6.15 kernel for this (like the one in our merge tree) but it appears to be The Way Of The Future, for now. You probably want something like VM_PFNMAP. The case for you will be slightly different, since you''re not mapping in device memory and won''t be using the usual IO remapping functions. A scan through the code might reveal some useful insight, since this has the look of a poorly documented interface - probably worth checking out the original changeset in kernel.org''s gitweb to see what changed together.> Mark Williamson''s blktap driver is calling > GNTTAB_unmap_grant_ref, but it''s unclear to me if the same unmap()''ing > problem exists there too.I can''t take credit for blktap, although I wish I could! IIRC, Andy Warfield wrote most of it. My name might have got into the copyright somewhere, but I''ve never worked on the tap functionality. As a result I also can''t give specific advice on the code, but I''d think it is probably a very good guide regarding mapping foreign pages into userspace. Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 25 Jan 2006, at 01:33, King, Steven R wrote:> To verify that I wasn''t making some new mistake, inspection of > linux/mm/mmap.c shows that Linux calls vma->close() *after* the page > table manipulation. In do_munmap(), the sequence is: > > detach_vmas_to_be_unmapped() > unmap_region() <--- page table manipulation buried in here > remove_vma_list() <--- vma->close() buried in hereAs others have suggested, you need a special type of vma that knows how to map/unmap/destruct ranges of granted mappings. A good way to do this would be to have gnttab.c export a device file that has an mmap() function. You can then add vma hook functions that know that any mapping created in that vma range must be destructed via gnt_unmap operations.> Is this turning into a Xen problem? Why does Xen forbid implicit > unmapping of shared pages?It''s hard for Xen to distinguish between mappings created via grants vs. mappings created because domain X is privileged to be able to map any page of domain Y. So far we''ve only really had grant-mapping code in the kernel. Doing it in user space just needs some work on the clean-up path. We expect guests to track and clean up their own mess. :-) -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 1/24/06, King, Steven R <steven.r.king@intel.com> wrote:> Vma->close() didn''t work. Thanks for the great suggestion though. :^)I''ve run up against this in the past. Linux vmas don''t have a hook point that allows you to clean things up before the associated PTEs are zapped, and so you are forced to take the implicit unmap path if you unexpectedly destroy an mmaped region containing grants without cleaning them up properly. One solution would be to add a new vma destructor op that gets called prior to the zap and then let your driver clean things up there. I have no idea if this would fly with the LKML folk though. ;) Fixing the implicit unmap code to automagically clean up on (e.g.) vma destruction is not the correct solution. There isn''t enough information to associate a mapped PTE to the appropriate active grant, and i don''t think we want to add the space overhead in Xen required to track this sort of thing just to babysit VMs that don''t clean things up correctly. If a grant is mapped multiple times, the implicit unmap code has no way to tell which active grant to garbage collect, and things can go terribly wrong. As other people have said, grants to user space are rather less thoroughly tested, but an extra vma op seems to me to be a good solution. a. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel