What''s the standard practice regarding freeing xenheap pages that have been shared with dom0 (and may currently be mapped)? Do we need to make sure they''re unmapped before we free them, or can we just call free_xenheap_pages() and let the reference counting take care of actually freeing them when dom0 unmaps them (if ever)? I''m looking at allowing xentrace buffers to be resized, and need to deal with potentially freeing pages which may be mapped by dom0. If it''s as simple as free-and-forget, I''ll include that functionality; if it''s more complicated, I''ll punt until another time (as I really only care about increasing the size of each cpu''s buffer, not resizing buffers). -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> George Dunlap <George.Dunlap@eu.citrix.com> 18.12.09 15:04 >>> >What''s the standard practice regarding freeing xenheap pages that have >been shared with dom0 (and may currently be mapped)? Do we need to >make sure they''re unmapped before we free them, or can we just call >free_xenheap_pages() and let the reference counting take care of >actually freeing them when dom0 unmaps them (if ever)?No, free_xenheap_pages() is what would be called when the refcount drops to zero. What you need to do is, afaics, a simply put_page_and_type(), which undoes the one type and one general reference share_xen_page_with_guest() installed. This will then either free the page right away (if Dom0 doesn''t have it mapped) or defer freeing until when Dom0 unmaps it. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 18/12/2009 14:29, "Jan Beulich" <JBeulich@novell.com> wrote:>>>> George Dunlap <George.Dunlap@eu.citrix.com> 18.12.09 15:04 >>> >> What''s the standard practice regarding freeing xenheap pages that have >> been shared with dom0 (and may currently be mapped)? Do we need to >> make sure they''re unmapped before we free them, or can we just call >> free_xenheap_pages() and let the reference counting take care of >> actually freeing them when dom0 unmaps them (if ever)? > > No, free_xenheap_pages() is what would be called when the refcount > drops to zero. What you need to do is, afaics, a simply > put_page_and_type(), which undoes the one type and one general > reference share_xen_page_with_guest() installed. This will then > either free the page right away (if Dom0 doesn''t have it mapped) or > defer freeing until when Dom0 unmaps it.In fact not. put_page() calls free_domheap_page() when the refcnt falls to zero. And free_domheap_page() does not actually free Xen-heap pages: it is expected that this will happen explicitly later (e.g., when domain is destroyed) -- this is to deal with Xen itself possibly still using those pages, and those uses are not reference counted. I think the right thing to do would be very roughly: On allocation: page = alloc_xenheap_page(); share_xen_page_with_guest(page); On deallocation: if (test_and_clear(PGC_allocated)) put_page(); if (page->count_info & PGC_count_mask) return -EBUSY; free_xenheap_page(page); -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser wrote:> I think the right thing to do would be very roughly: > On allocation: > page = alloc_xenheap_page(); > share_xen_page_with_guest(page); > On deallocation: > if (test_and_clear(PGC_allocated)) put_page(); > if (page->count_info & PGC_count_mask) return -EBUSY; > free_xenheap_page(page); >Hmm, the first two should probably happen in the opposite order, no? I.e., we should check to see that for each page we want to deallocate, page->count_info & PGC_count_mask == 0, returning -EBUSY if that fails; then unmark allocate and call free xenheap page? -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> Keir Fraser <keir.fraser@eu.citrix.com> 18.12.09 16:24 >>> >I think the right thing to do would be very roughly: >On allocation: > page = alloc_xenheap_page(); > share_xen_page_with_guest(page); >On deallocation: > if (test_and_clear(PGC_allocated)) put_page(); > if (page->count_info & PGC_count_mask) return -EBUSY;Hmm, and how would you recover from that? I don''t think George wants his enclosing operation to fail because of a page still being mapped. On x86-64, at least, it might be possible to play with PGC_xen_heap, to convert a Xen heap page to a domain heap one, but that wouldn''t cover anyone else (and would seem rather hackish, if it works at all).> free_xenheap_page(page);Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 18/12/2009 15:29, "George Dunlap" <George.Dunlap@eu.citrix.com> wrote:> Hmm, the first two should probably happen in the opposite order, no? > I.e., we should check to see that for each page we want to deallocate, > page->count_info & PGC_count_mask == 0, returning -EBUSY if that fails; > then unmark allocate and call free xenheap page?No, PGC_allocated holds a reference count. Reference count cannot be zero while PGC_allocated is true. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 18/12/2009 15:44, "Jan Beulich" <JBeulich@novell.com> wrote:>> On deallocation: >> if (test_and_clear(PGC_allocated)) put_page(); >> if (page->count_info & PGC_count_mask) return -EBUSY; > > Hmm, and how would you recover from that? I don''t think George wants > his enclosing operation to fail because of a page still being mapped. On > x86-64, at least, it might be possible to play with PGC_xen_heap, to > convert a Xen heap page to a domain heap one, but that wouldn''t > cover anyone else (and would seem rather hackish, if it works at all).I assume he would wrap it all up in a retryable hypercall. E.g., I don''t know the scenario we''re thinking about, but perhaps xentrace tools go bad and leak mappings, so the hypercall fails. So you restart xentrace tools, and now the hypercall would work on next attempt. You wouldn''t have to be stuck forever, but yes you are stuck, or at least have to keep tracking that page in Xen, until dom0 stops mapping the page, obviously! -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich wrote:>>>> Keir Fraser <keir.fraser@eu.citrix.com> 18.12.09 16:24 >>> >>>> >> I think the right thing to do would be very roughly: >> On allocation: >> page = alloc_xenheap_page(); >> share_xen_page_with_guest(page); >> On deallocation: >> if (test_and_clear(PGC_allocated)) put_page(); >> if (page->count_info & PGC_count_mask) return -EBUSY; >> > > Hmm, and how would you recover from that? I don''t think George wants > his enclosing operation to fail because of a page still being mapped. On > x86-64, at least, it might be possible to play with PGC_xen_heap, to > convert a Xen heap page to a domain heap one, but that wouldn''t > cover anyone else (and would seem rather hackish, if it works at all). >I wouldn''t really mind the op failing; I just want it to fail cleanly. All-or-nothing would be optimal IMHO. Failing such that everything will automatically clean itself up if/when dom0 does unmap the pages would be OK too. Having half the pages freed / marked unallocated, but some still allocated doesn''t seem very good. Anyway, I care more about allocating larger buffers than I do about re-allocating, so maybe I''ll just punt on freeing maybe-mapped buffers for now. However, that brings up another point: suppose that I do several allocations, one per cpu, and one fails. Now I want to go and free the buffers I''ve just allocated (which shouldn''t be mapped in dom0 yet). Can I not just call free_xenheap_pages()? Do I need to clear PGC_allocated before calling? Is any of this stuff written down somewhere? :-) -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> George Dunlap <george.dunlap@eu.citrix.com> 18.12.09 17:19 >>> >However, that brings up another point: suppose that I do several >allocations, one per cpu, and one fails. Now I want to go and free the >buffers I''ve just allocated (which shouldn''t be mapped in dom0 yet). >Can I not just call free_xenheap_pages()? Do I need to clear >PGC_allocated before calling?If you would fully trust Dom0 (including that it will never run malicious code), then you could do so I think. But security people may not like that, so it would still seem be better to verify by clearing the allocated flag, putting the page, and checking its refcount before freeing.>Is any of this stuff written down somewhere? :-)I''m afraid it isn''t. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 18/12/2009 16:19, "George Dunlap" <George.Dunlap@eu.citrix.com> wrote:> However, that brings up another point: suppose that I do several > allocations, one per cpu, and one fails. Now I want to go and free the > buffers I''ve just allocated (which shouldn''t be mapped in dom0 yet). > Can I not just call free_xenheap_pages()? Do I need to clear > PGC_allocated before calling?Once you have called share_xen_page_with_guest(), dom0 can map that page at any time and it is not safe to free_xenheap_page() without first checking for guest mappings. Perhaps you can lazily adjust tarce buffers downwards in size? I.e. Have a go, remember if you fail, and try again on future buffer-adjustment hypercalls?> Is any of this stuff written down somewhere? :-)No. :-) -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Dec 18, 2009 at 4:40 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:> Once you have called share_xen_page_with_guest(), dom0 can map that page at > any time and it is not safe to free_xenheap_page() without first checking > for guest mappings. > > Perhaps you can lazily adjust tarce buffers downwards in size? I.e. Have a > go, remember if you fail, and try again on future buffer-adjustment > hypercalls?I don''t expect buffer-adjustment to be a common thing, so the "lazy" option might end up being incredibly lazy. :-) My plan now is to do alloc-once only (i.e., keeping the same semantics), and make all the allocations before sharing the page with dom0. In the future, I may add the tear-down and check for reference count, with an "undo" if there are still outstanding reference counts. I think the most likely reason to run into this scenario is if someone accidentally starts xentrace or xenmon when there''s already an instance of one of them running, in which case the right thing to do is fail the new one and let the user decide whether to kill the old process or not. -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel