Scheer, Roque
2004-May-20 13:18 UTC
[Xen-devel] Seamlessly sharing identical memory pages among domains
I''d like to know if XEN is or will be implementing any mechanism for transparently sharing pages with identical content among domains, to minimize memory consumption, similar to what is implemented in VMWare''s ESX server (chapter 4 of the paper whose link is below). http://www.waldspurger.org/carl/papers/esx-mem-osdi02.pdf Thanks, - Roque ------------------------------------------------------- This SF.Net email is sponsored by: Oracle 10g Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we''ll give you the exam FREE. http://ads.osdn.com/?ad_id149&alloc_id66&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Ian Pratt
2004-May-21 01:17 UTC
Re: [Xen-devel] Seamlessly sharing identical memory pages among domains
> I''d like to know if XEN is or will be implementing any mechanism for > transparently sharing pages with identical content among domains, to > minimize memory consumption, similar to what is implemented in VMWare''s > ESX server (chapter 4 of the paper whose link is below).I''m not convinced there''s that much sharing to be had between VMs: One side effect of implementing some test code for live migration was that I collected some traces containing crc fingerprints of the memory pages of several domains. There weren''t many hashes in common. Perhaps this shouldn''t be surprising -- application text is typically small compared to the heap data they operate on. The actual datasets operated on by domains are typically distinct. However, there might be some value in having a shared read-only buffer cache, if only for the saved IO ops. Implementing a cache that replicates pages in memory (rather than sharing) is trivial. Doing a proper shared cache is slightly trickier given the paravirtualised memory interface -- we''d have to introduce guests to a new kind of write fault: "A write fault has occurred, and you''ll have to copy the page because this machine page is immutable as it is already shared with other domains". Modifying Linux to handle this wouldn''t be hard. Of course, if you''re running a domain on Xen''s shadow page tables then all of this manipulation is easy and can be done transparently to the domain. Of course, you pay a performance penalty... Ian ------------------------------------------------------- This SF.Net email is sponsored by: Oracle 10g Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we''ll give you the exam FREE. http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Kip Macy
2004-May-21 03:58 UTC
Re: [Xen-devel] Seamlessly sharing identical memory pages among domains
> > Doing a proper shared cache is slightly trickier given the > paravirtualised memory interface -- we''d have to introduce guests > to a new kind of write fault: "A write fault has occurred, and > you''ll have to copy the page because this machine page is > immutable as it is already shared with other domains". Modifying > Linux to handle this wouldn''t be hard.Would this just involve adding an extra error code to what x86 already uses and then modifying the guest to understand that that error code means that the page has to be COWed? -Kip ------------------------------------------------------- This SF.Net email is sponsored by: Oracle 10g Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we''ll give you the exam FREE. http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Ian Pratt
2004-May-21 06:56 UTC
Re: [Xen-devel] Seamlessly sharing identical memory pages among domains
> > > > Doing a proper shared cache is slightly trickier given the > > paravirtualised memory interface -- we''d have to introduce guests > > to a new kind of write fault: "A write fault has occurred, and > > you''ll have to copy the page because this machine page is > > immutable as it is already shared with other domains". Modifying > > Linux to handle this wouldn''t be hard. > > Would this just involve adding an extra error code to what x86 already > uses and then modifying the guest to understand that that error code > means that the page has to be COWed?Yep. The only slight subtlety is that that the page in question may appear in multiple page tables and at multiple VAs: if the page is truly read-write shared in the guest (e.g. sysv shared memory [*]) then it it will be necessary to find all the ptes and patch them to the newly allocated machine frame. I think such pages are rare, and tracking them down by scanning the list of vma''s that are mapping the object (e.g. a file) in question is pretty easy and quick. However, this would most sensibly be done by making minor modifications to architecture independent code, which is something we try to avoid. I guess we could just have arch xen grub its way through the higher-level data structures... Ian [*] Hmm, do memory mapped files behave like this too? I can''t remember. ------------------------------------------------------- This SF.Net email is sponsored by: Oracle 10g Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we''ll give you the exam FREE. http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Kip Macy
2004-May-21 11:21 UTC
Re: [Xen-devel] Seamlessly sharing identical memory pages among domains
> > Yep. The only slight subtlety is that that the page in question > may appear in multiple page tables and at multiple VAs: if the > page is truly read-write shared in the guest (e.g. sysv shared > memory [*]) then it it will be necessary to find all the ptes and > patch them to the newly allocated machine frame. I think such > pages are rare, and tracking them down by scanning the list of > vma''s that are mapping the object (e.g. a file) in question is > pretty easy and quick. However, this would most sensibly be done > by making minor modifications to architecture independent code,Just adding an upcall wouldn''t be too terrible. Iterating through the list of vtop mappings for a page should be fairly light weight. Politically, the most realistic thing to do is put the arch-independent code changes in the xen arch-dependent tree wherever possible. I don''t think anyone wants to see xen specific changes going on outside of the arch subtree.> [*] Hmm, do memory mapped files behave like this too? I can''t > remember.If multiple processes share the page read-write, the OS has to have some way to know about it. -Kip ------------------------------------------------------- This SF.Net email is sponsored by: Oracle 10g Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we''ll give you the exam FREE. http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Scheer, Roque
2004-May-21 11:56 UTC
RE: [Xen-devel] Seamlessly sharing identical memory pages among domains
>> I''m not convinced there''s that much sharing to be had between >> VMsIn the paper Carl Waldspurger reports some worthwhile sharing under certain real world workloads: | Total| Shared |Reclaimed | Guest Types| MB | MB | % | MB | % | -----------+------+----+-----+-----+------+ A)10 WinNT | 2048 | 880| 42.9| 673 | 32.9 | B) 9 Linux | 1846 | 539| 29.2| 345 | 18.7 | C) 5 Linux | 1658 | 165| 10.0| 120 | 7.2 |>> Doing a proper shared cache is slightly trickier given theparavirtualised>> memory interface -- we''d have to introduce guests to a new kind ofwrite>> faultWouldn''t it be possible for XEN to share the pages tranparently, without the guest OSes needing to know it? - Roque ------------------------------------------------------- This SF.Net email is sponsored by: Oracle 10g Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we''ll give you the exam FREE. http://ads.osdn.com/?ad_id149&alloc_id66&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Ian Pratt
2004-May-21 12:05 UTC
Re: [Xen-devel] Seamlessly sharing identical memory pages among domains
> In the paper Carl Waldspurger reports some worthwhile sharing under > certain real world workloads: > > | Total| Shared |Reclaimed | > Guest Types| MB | MB | % | MB | % | > -----------+------+----+-----+-----+------+ > A)10 WinNT | 2048 | 880| 42.9| 673 | 32.9 | > B) 9 Linux | 1846 | 539| 29.2| 345 | 18.7 | > C) 5 Linux | 1658 | 165| 10.0| 120 | 7.2 |I haven''t got any figures for WinXP. The amount of sharing for Linux instances will be very work load dependent. I''d bet that almost all of it could be had through a shared buffer cache (from a CoW file system).> Wouldn''t it be possible for XEN to share the pages tranparently, without > the guest OSes needing to know it?Only for guests running in "shadow page table" mode, otherwise the guest knows about real machine frame numbers. Ian ------------------------------------------------------- This SF.Net email is sponsored by: Oracle 10g Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we''ll give you the exam FREE. http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Steven Hand
2004-May-21 12:08 UTC
Re: [Xen-devel] Seamlessly sharing identical memory pages among domains
> >> I''m not convinced there''s that much sharing to be had between > >> VMs > > In the paper Carl Waldspurger reports some worthwhile sharing under > certain real world workloads: > > | Total| Shared |Reclaimed | > Guest Types| MB | MB | % | MB | % | > -----------+------+----+-----+-----+------+ > A)10 WinNT | 2048 | 880| 42.9| 673 | 32.9 | > B) 9 Linux | 1846 | 539| 29.2| 345 | 18.7 | > C) 5 Linux | 1658 | 165| 10.0| 120 | 7.2 |Yes; and we may do some shared text support at some point. Not clear how much additional benefit comes from the "scanning" they did; a simpler approach to detect read-only sharing is to explicitly check for resident versions of a certain text segment across VMs. Having some "look-aside" in the block device interface (partic if also doing sharing at that level as per our current CoW implementation) might get considerable benefit for little cost.> >> Doing a proper shared cache is slightly trickier given the > >> paravirtualised memory interface -- we''d have to introduce guests > >>to a new kind of write fault > > Wouldn''t it be possible for XEN to share the pages tranparently, without > the guest OSes needing to know it?Possibly. One challenge here is to maintain partitioning (hard guarantees) while providing sharing. Depending on how important isolation is to you, this is relatively less or more difficult. Something we''ll be considering post OSDI anyway.. cheers, S. ------------------------------------------------------- This SF.Net email is sponsored by: Oracle 10g Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we''ll give you the exam FREE. http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel