Hi, I''m using Xen 2.0 (base kernel is 2.6.8.1) and found that it doesn''t enable the shadow page tables in Xen though there is code there. If the shadow page tables are enabled, does it mean that the guest Linux'' memory management module could be fully virtualized? Thanks, Chengyuan ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Currently, shadow p.t.''s are only enabled when migrating OSes between machines. This allows us to copy the memory image while the OS is still running, and track which pages are updated after being copied (and so need to be recopied). We intend to flesh out the shadow p.t. code a little mroe to support full memory virtualisation. It''s not there yet, but it won''t require an enormous amount of code to get it going. -- Keir> Hi, > > I''m using Xen 2.0 (base kernel is 2.6.8.1) and found that it doesn''t > enable the shadow page tables in Xen though there is code there. > > If the shadow page tables are enabled, does it mean that the guest > Linux'' memory management module could be fully virtualized? > > Thanks, > Chengyuan > > > ------------------------------------------------------- > This SF.net email is sponsored by: IT Product Guide on ITManagersJournal > Use IT products in your business? Tell us what you think of them. Give us > Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more > http://productguide.itmanagersjournal.com/guidepromo.tmpl > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xen-devel------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
If the shadow pagetable is enabled, Xen will maintain pagetables for every guest process. If 1000 processes are running in the guest, xen will at least allocate 1000 pages as shadow pagetables. Will it consume too many memory frames? On Fri, 08 Oct 2004 10:11:18 +0100, Keir Fraser <keir.fraser@cl.cam.ac.uk> wrote:> > Currently, shadow p.t.''s are only enabled when migrating OSes between > machines. This allows us to copy the memory image while the OS is > still running, and track which pages are updated after being copied > (and so need to be recopied). > > We intend to flesh out the shadow p.t. code a little mroe to support > full memory virtualisation. It''s not there yet, but it won''t require > an enormous amount of code to get it going. > > -- Keir > > > > > Hi, > > > > I''m using Xen 2.0 (base kernel is 2.6.8.1) and found that it doesn''t > > enable the shadow page tables in Xen though there is code there. > > > > If the shadow page tables are enabled, does it mean that the guest > > Linux'' memory management module could be fully virtualized? > > > > Thanks, > > Chengyuan > > > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: IT Product Guide on ITManagersJournal > > Use IT products in your business? Tell us what you think of them. Give us > > Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more > > http://productguide.itmanagersjournal.com/guidepromo.tmpl > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/xen-devel > >------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> If the shadow pagetable is enabled, Xen will maintain pagetables forevery guest> process. If 1000 processes are running in the guest, xen will at leastallocate 1000> pages as shadow pagetables. Will it consume too many memory frames?That''s the downside of shadow pagetables, and hence why we prefer to run guests in ''direct mode'' unless we''re doing something ''weird'' to them - e.g. when we migrate a domain we switch to shadow page tables for the duration of the migration. Apache 1.3 is a good example of an application that requires large amounts of memory for pagetables (many thousands of pages), and with shadow pagetables this requiment effectively doubles. The shadow pagetables are just a cache, so its simple to evict pages if you run low on memory, but then performance will suck big time. Ian On Fri, 08 Oct 2004 10:11:18 +0100, Keir Fraser <keir.fraser@cl.cam.ac.uk> wrote:> > Currently, shadow p.t.''s are only enabled when migrating OSes between > machines. This allows us to copy the memory image while the OS is > still running, and track which pages are updated after being copied > (and so need to be recopied). > > We intend to flesh out the shadow p.t. code a little mroe to support > full memory virtualisation. It''s not there yet, but it won''t require > an enormous amount of code to get it going. > > -- Keir > > > > > Hi, > > > > I''m using Xen 2.0 (base kernel is 2.6.8.1) and found that it doesn''t> > enable the shadow page tables in Xen though there is code there. > > > > If the shadow page tables are enabled, does it mean that the guest > > Linux'' memory management module could be fully virtualized? > > > > Thanks, > > Chengyuan > > > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: IT Product Guide on > > ITManagersJournal Use IT products in your business? Tell us what you> > think of them. Give us Your Opinions, Get Free ThinkGeek Gift > > Certificates! Click to find out more > > http://productguide.itmanagersjournal.com/guidepromo.tmpl > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/xen-devel > >------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Fri, Oct 08, 2004 at 10:11:18AM +0100, Keir Fraser wrote:> We intend to flesh out the shadow p.t. code a little mroe to support > full memory virtualisation. It''s not there yet, but it won''t require > an enormous amount of code to get it going.Any idea what the time frame for this is? I was considering taking a stab at it myself, but don''t want to duplicate work. My longer term goal is to try to get copy-on-write sharing of memory pages between domains and to see how far Xen can scale in running many nearly-identical virtual machines. To implement this, however, will require (at least as I''ve thought it through) memory virtualization. So that was the first thing I was planning to work on. If no one else was actively working on this, I''m happy to discuss the design and contribute what I end up with. (Both for the memory virtualization and the copy-on-write work.) (This is also the reason for my interest in copy-on-write disks, though it''s looking like LVM may be too heavyweight for supporting large numbers of VMs.) --Michael Vrable ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> (This is also the reason for my interest in copy-on-write disks, though > it''s looking like LVM may be too heavyweight for supporting large > numbers of VMs.)Hi Michael, I''d be interested to know what sort of overhead results you''ve seen on lvm -- I''ve been looking at block devices myself a bit lately and if lvm doesn''t look like it can be made to scale as a solution for cow, I might be able to arrange some sort of alternative. Any sort of summary of your lvm thoughts/experiences would be great. best, andy. ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Tue, Oct 12, 2004 at 09:09:24PM +0100, Andrew Warfield wrote:> I''d be interested to know what sort of overhead results you''ve seen > on lvm -- I''ve been looking at block devices myself a bit lately and > if lvm doesn''t look like it can be made to scale as a solution for > cow, I might be able to arrange some sort of alternative. Any sort of > summary of your lvm thoughts/experiences would be great.My primary concern with LVM snapshot scalability is kernel memory consumption. (I haven''t done any performance benchmarking.) For each LVM snapshot that is created, the kernel allocates: - 1 MB of pages for a kcopyd_client. - Up to a 2 MB hash table for tracking exceptions (areas of the disk that have been changed). - The exceptions themselves. (Not very large individually, but it all has to be in kernel memory--on the order of 16 bytes per area of disk [say ~16 K] that has been remapped, so that could grow to 1 MB of memory per 1 GB COW disk.) (These numbers are gathered from looking through the snapshot code.) The kcopyd_client can (I think) be safely shared between snapshots--I patched the kernel to do this, and it worked fine under the limited testing I gave it. So this issue could be worked around. But the other two are more troublesome, since the assumption that all the COW mapping tables available in kernel memory can''t be changed without a lot of work. The problem is currently worse than it has to be since the LVM code won''t sleep if the kernel memory isn''t immediately available; it errors out. Even with that fixed (which may happen eventually: http://www.redhat.com/archives/dm-devel/2004-January/msg00079.html), LVM is still pinning on the order of megabytes of kernel memory per snapshot. That is fine for some workloads, but I''d like to see if I can get to hundreds or thousands of COW disks--in which case this is too much memory per disk. I''m currently wondering if a solution implemented partially in userspace is viable--put the COW driver in userspace, let the on-disk data structures get cached in the page cache, but any COW disks not being actively utilized would have data swapped out of memory. Perhaps communicate to the kernel via the nbd interface or something similar? I haven''t put much thought into this yet. I expect performance would be worse than LVM, but that''s a tradeoff I''m willing to make. --Michael Vrable ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Hello Some time ago you talked about copy-on-write memory to enable large numbers of nearly identical machines to run on the same physical hardware. I have also thought about that - after all equivalent mechansisms already exist to provide copy-on-write semantics for process forking (and also I think in the /proc/mm mechanisms used in some versions of UML separate kernel address space (SKAS) handling). Have you advanced along this path, or is anyone actively doing that? The problem you found the other day suggests that you were looking pretty closely at code that relates to this question. My very crude understanding is that one would want to share the physical memory allocation that is currently given to a single xenU domain between a group of xenU domains each of which has its own copy-on-write mapping that starts from a read-only image that is shared between all of them. The table grant mehanism would need to understand the relationship between members of such a group of domains. But as I say, this is still very hazy to me. On the other point you raised, about copy-on-write filesystem images, I looked at evms2 (http://evms.sourceforge.net/), which sits above lvm2 and other systems and offers its own snapshot feature. It is only usable on disks that it controls, so you either have to use a separate disk from the one you boot from, or you have to use an initrd to be able to boot from an evms volume. The snapshot feature seemed ''snappier'' (I was able to create a number of snapshot objects and activate them without memory problems of the kind that I encountered with lvm2 snapshots). However it didn''t seem possible to make it provide multiple persistent writable snapshots of the kind that would support multiple xenU domains where each uses its own copy-on-write image of an underlying readonly root filesystem. Did I miss something? Has anyone else tried using evms for these purposes? I saw Ian''s comments about gnbd and csnap (http://sources.redhat.com/cluster/), but it seems that the csnap mechanism is in a pretty early state of development. Regards Peri Michael Vrable wrote: Michael Vrable wrote:>On Fri, Oct 08, 2004 at 10:11:18AM +0100, Keir Fraser wrote: > > >>We intend to flesh out the shadow p.t. code a little mroe to support >>full memory virtualisation. It''s not there yet, but it won''t require >>an enormous amount of code to get it going. >> >> > >Any idea what the time frame for this is? I was considering taking a >stab at it myself, but don''t want to duplicate work. > >My longer term goal is to try to get copy-on-write sharing of memory >pages between domains and to see how far Xen can scale in running many >nearly-identical virtual machines. To implement this, however, will >require (at least as I''ve thought it through) memory virtualization. So >that was the first thing I was planning to work on. > >If no one else was actively working on this, I''m happy to discuss the >design and contribute what I end up with. (Both for the memory >virtualization and the copy-on-write work.) > >(This is also the reason for my interest in copy-on-write disks, though >it''s looking like LVM may be too heavyweight for supporting large >numbers of VMs.) > >--Michael Vrable > > >------------------------------------------------------- >This SF.net email is sponsored by: IT Product Guide on ITManagersJournal >Use IT products in your business? Tell us what you think of them. Give us >Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more >http://productguide.itmanagersjournal.com/guidepromo.tmpl >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/xen-devel > > > >------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Peri Hankey
2004-Oct-25 12:12 UTC
[Xen-devel] Copy-on-write memory to allow many more xenU domains per machine
Hello Some time ago Michael Vrable talked about copy-on-write memory to enable large numbers of nearly identical machines to run on the same physical hardware. Is this a feasible proposition? I asked a few days ago in the original thread, but no-one seems to have noticed. It seems to me that it would be a very significant feature to offer. Regards Peri ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Ian Pratt
2004-Oct-25 12:21 UTC
Re: [Xen-devel] Copy-on-write memory to allow many more xenU domains per machine
> Hello > > Some time ago Michael Vrable talked about copy-on-write memory to enable > large numbers of nearly identical machines to run on the same physical > hardware. Is this a feasible proposition? I asked a few days ago in the > original thread, but no-one seems to have noticed. It seems to me that > it would be a very significant feature to offer.It''s useful for honeypots and other situations where you want very large numbers of VMs, but isn''t generally a huge win. When I was doing the live migration work I recorded fingerprints of all VM pages that were on several systems, and didn''t find a whole lot of commonality between VMs. CoW sharing of memory is on the list of features we''d like to add to Xen, but implementing the shared buffer cache is rather higher on the todo list. Ian ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
David Hopwood
2004-Oct-25 18:24 UTC
Re: [Xen-devel] Copy-on-write memory to allow many more xenU domains per machine
Ian Pratt wrote:>>Hello >> >>Some time ago Michael Vrable talked about copy-on-write memory to enable >>large numbers of nearly identical machines to run on the same physical >>hardware. Is this a feasible proposition? I asked a few days ago in the >>original thread, but no-one seems to have noticed. It seems to me that >>it would be a very significant feature to offer. > > It''s useful for honeypots and other situations where you want > very large numbers of VMs, but isn''t generally a huge win. > > When I was doing the live migration work I recorded fingerprints > of all VM pages that were on several systems, and didn''t find a > whole lot of commonality between VMs.I thought the idea would be to have a ''clone'' primitive that would work in a similar way to Unix fork? The clone would start off sharing all pages, although a lot of them would get copied fairly quickly. Does anyone have an idea of what proportion or number of pages in a Linux system stay constant (and are not paged out) after boot? -- David Hopwood <david.nospam.hopwood@blueyonder.co.uk> ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Ronald G. Minnich
2004-Oct-25 19:52 UTC
Re: [Xen-devel] Copy-on-write memory to allow many more xenU domains per machine
On Mon, 25 Oct 2004, David Hopwood wrote:> I thought the idea would be to have a ''clone'' primitive that would work > in a similar way to Unix fork? The clone would start off sharing all > pages, although a lot of them would get copied fairly quickly. Doesyes but ... if most or almost most pages get copied, COW is a loss. Try to find that old paper somebody wrote about "bovaphobic" behaviour -- interesting numbers in there. COW is not always a good thing. ron ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Ian Pratt
2004-Oct-25 20:19 UTC
Re: [Xen-devel] Copy-on-write memory to allow many more xenU domains per machine
> I thought the idea would be to have a ''clone'' primitive that would work > in a similar way to Unix fork? The clone would start off sharing all > pages, although a lot of them would get copied fairly quickly. Does > anyone have an idea of what proportion or number of pages in a Linux > system stay constant (and are not paged out) after boot?We''ve already done some work on implementing very lightweight VM checkpoints, for purposes of debugging and s/w fault tolerance. ''VM clone'' is on the todo list, but I think it''ll be a while before this makes it into a stable release. Ian ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Michael Vrable
2004-Oct-25 22:38 UTC
Re: [Xen-devel] Copy-on-write memory to allow many more xenU domains per machine
On Mon, Oct 25, 2004 at 01:12:22PM +0100, Peri Hankey wrote:> Some time ago Michael Vrable talked about copy-on-write memory to enable > large numbers of nearly identical machines to run on the same physical > hardware. Is this a feasible proposition? I asked a few days ago in the > original thread, but no-one seems to have noticed. It seems to me that > it would be a very significant feature to offer.I''m still around, though a bit slow to respond. Ian mentioned that honeypots are one the scenarios where copy-on-write sharing of memory makes a lot of sense; that''s the reason that I''m trying implement it. I hope, of course, that if I do manage to get something working it would be useful in other scenarios as well. My current status: I''ve been working first on using shadow page tables to hide the actual machine addresses of pages from domains. I''m only partway through doing that, and may yet go back and change how I''m approaching that (I''m still learning bits about how Xen works as I''m going through to modify the code; I''m also thinking now of changes that would be good to make copy-on-write easier later). I''m trying to write up some of my design thoughts; if anyone here is interested or would like to offer comments, I can post that here. David mentioned implementing a clone() for VMs; that is something that I''d also like to try out. (It should be possible to test that after only the first step of hiding machine addresses, before copy-on-write is implemented, though copy-on-write will make it efficient.) If someone else beats me to getting this implemented, I suppose I can''t complain. But if this is an area that isn''t getting much current work, and I''m able to contribute (a hopefully clean implementation that others wouldn''t mind using), that would be great. --Michael Vrable ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Peri Hankey
2004-Oct-26 08:15 UTC
Re: [Xen-devel] Copy-on-write memory to allow many more xenU domains per machine
I hadn''t been thinking of the honeypot scenario myself, but of the plain old ''more virtual computers than you would have thought possible'' scene.>From Ian''s account of it there''s less scope for that than I had imagined.I can be heavy slow myself (''heavy'' is apparently Glaswgian slang for ''very'' - think ''heavy light'') - especially when immersed in the technical undergrowth. Regards Peri Michael Vrable wrote:>On Mon, Oct 25, 2004 at 01:12:22PM +0100, Peri Hankey wrote: > > >>Some time ago Michael Vrable talked about copy-on-write memory to enable >>large numbers of nearly identical machines to run on the same physical >>hardware. Is this a feasible proposition? I asked a few days ago in the >>original thread, but no-one seems to have noticed. It seems to me that >>it would be a very significant feature to offer. >> >> > >I''m still around, though a bit slow to respond. > >Ian mentioned that honeypots are one the scenarios where copy-on-write >sharing of memory makes a lot of sense; that''s the reason that I''m >trying implement it. I hope, of course, that if I do manage to get >something working it would be useful in other scenarios as well. > >My current status: I''ve been working first on using shadow page tables >to hide the actual machine addresses of pages from domains. I''m only >partway through doing that, and may yet go back and change how I''m >approaching that (I''m still learning bits about how Xen works as I''m >going through to modify the code; I''m also thinking now of changes that >would be good to make copy-on-write easier later). I''m trying to write >up some of my design thoughts; if anyone here is interested or would >like to offer comments, I can post that here. > >David mentioned implementing a clone() for VMs; that is something that >I''d also like to try out. (It should be possible to test that after >only the first step of hiding machine addresses, before copy-on-write is >implemented, though copy-on-write will make it efficient.) > >If someone else beats me to getting this implemented, I suppose I can''t >complain. But if this is an area that isn''t getting much current work, >and I''m able to contribute (a hopefully clean implementation that others >wouldn''t mind using), that would be great. > >--Michael Vrable > > >------------------------------------------------------- >This SF.net email is sponsored by: IT Product Guide on ITManagersJournal >Use IT products in your business? Tell us what you think of them. Give us >Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more >http://productguide.itmanagersjournal.com/guidepromo.tmpl >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/xen-devel > > > >------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Michael Vrable
2004-Oct-26 18:30 UTC
Re: [Xen-devel] Copy-on-write memory to allow many more xenU domains per machine
On Tue, Oct 26, 2004 at 09:15:05AM +0100, Peri Hankey wrote:> I hadn''t been thinking of the honeypot scenario myself, but of the plain > old ''more virtual computers than you would have thought possible'' scene. > From Ian''s account of it there''s less scope for that than I had imagined. > > I can be heavy slow myself (''heavy'' is apparently Glaswgian slang for > ''very'' - think ''heavy light'') - especially when immersed in the > technical undergrowth."Memory Resource Management in VMware ESX Server" by Carl Waldspurger has some numbers on memory sharing between VMs: http://www.usenix.org/events/osdi02/tech/waldspurger/waldspurger.pdf (This paper is also cited in the Xen SOSP 2003 paper.) Sharing between identical machines is good, though sharing between dissimilar machines is not as high as I remembered (copy-on-write sharing is able to save around 7-30% of memory in real-world testing). --Michael Vrable ------------------------------------------------------- This SF.Net email is sponsored by: Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader''s Choice Award Winner for best database on Linux. http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Ian Pratt
2004-Oct-26 19:43 UTC
Re: [Xen-devel] Copy-on-write memory to allow many more xenU domains per machine
> "Memory Resource Management in VMware ESX Server" by Carl Waldspurger > has some numbers on memory sharing between VMs: > http://www.usenix.org/events/osdi02/tech/waldspurger/waldspurger.pdf > (This paper is also cited in the Xen SOSP 2003 paper.) > > Sharing between identical machines is good, though sharing between > dissimilar machines is not as high as I remembered (copy-on-write > sharing is able to save around 7-30% of memory in real-world testing).As I recall, Carl''s results were on Windows, which might explain why I was seeing rather less. Linux resident set sizes tend to be smaller, and there''s generally rather more diversity in the install base (at least around here). Ian ------------------------------------------------------- This SF.Net email is sponsored by: Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader''s Choice Award Winner for best database on Linux. http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Michael Vrable
2004-Oct-26 22:19 UTC
Re: [Xen-devel] Copy-on-write memory to allow many more xenU domains per machine
On Tue, Oct 26, 2004 at 08:43:30PM +0100, Ian Pratt wrote:> > "Memory Resource Management in VMware ESX Server" by Carl Waldspurger > > has some numbers on memory sharing between VMs: > > http://www.usenix.org/events/osdi02/tech/waldspurger/waldspurger.pdf > > (This paper is also cited in the Xen SOSP 2003 paper.) > > > > Sharing between identical machines is good, though sharing between > > dissimilar machines is not as high as I remembered (copy-on-write > > sharing is able to save around 7-30% of memory in real-world testing). > > As I recall, Carl''s results were on Windows, which might explain > why I was seeing rather less. Linux resident set sizes tend to be > smaller, and there''s generally rather more diversity in the > install base (at least around here).Both were tested in that paper; Windows was at the high end of the scale and Linux at the low end: Reproducing Figure 5 from the paper in slightly abbreviated form: Guests % Shared % Reclaimed 10 x WinNT 42.9 32.9 9 x Linux 29.2 18.7 5 x Linux 10.0 7.2 "% Shared" is the fraction of pages which are shared; "% Reclaimed" measures how much memory sharing the pages is able to save. Three systems is still a limited test set. --Michael Vrable ------------------------------------------------------- This SF.Net email is sponsored by: Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader''s Choice Award Winner for best database on Linux. http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Jacob Gorm Hansen
2004-Nov-02 20:55 UTC
Re: [Xen-devel] Copy-on-write memory to allow many more xenU domains per machine
Michael Vrable wrote: > My current status: I''ve been working first on using shadow page tables > to hide the actual machine addresses of pages from domains. I''m only > partway through doing that, and may yet go back and change how I''m > approaching that (I''m still learning bits about how Xen works as I''m > going through to modify the code; I''m also thinking now of changes that > would be good to make copy-on-write easier later). I''m trying to write > up some of my design thoughts; if anyone here is interested or would > like to offer comments, I can post that here. > > David mentioned implementing a clone() for VMs; that is something that > I''d also like to try out. (It should be possible to test that after > only the first step of hiding machine addresses, before copy-on-write is > implemented, though copy-on-write will make it efficient. What about having a special variant of the pte-update operation, which takes as input a pointer to a pte-entry, and will change that entry into a read-only mapping of the same page contents, though perhaps with a different frame number in the pte. Then Xen can lookup the page contents in a global hashtable or binary tree (which might be faster, because you don''t need to scan the entire page contents unless you have a match), and see if there is a matching copy somewhere. If there is, it can decrease a refcount for the page provided by the user, change the mapping to point to the matching page, and free the user''s page if the refcount has reached zero. In this way, domains become responsible for sharing memory across the system, but only pages that are likely candidates for sharing (kernel text, user process text) are actually scanned, and you don''t get the overhead of shadow page tables. Jacob ------------------------------------------------------- This SF.Net email is sponsored by: Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader''s Choice Award Winner for best database on Linux. http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Ian Pratt
2004-Nov-02 21:33 UTC
Re: [Xen-devel] Copy-on-write memory to allow many more xenU domains per machine
> What about having a special variant of the pte-update operation, which > takes as input a pointer to a pte-entry, and will change that entry into > a read-only mapping of the same page contents, though perhaps with a > different frame number in the pte.I''m still not convinced we wouldn''t get most of the benefit by just sharing pages read from the block/file system, and hence avoiding all the hashing. Collecting this data is actually very easy -- its just a case of coming up with a few "realistic usage scenarios" in which to collect the data. We''d be happy to work with someone to do this... Ian ------------------------------------------------------- This SF.Net email is sponsored by: Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader''s Choice Award Winner for best database on Linux. http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
urmk@reason.marist.edu
2004-Nov-02 21:50 UTC
Re: [Xen-devel] Copy-on-write memory to allow many more xenU domains per machine
Check the s/390 port of the kernel for the XIP2 filesystem. It exploits the z/VM shared memory system to make a read-only ext2 like filesystem that is shared amonst all the guests - and uses execute-in-place. Any binaries on the xip2 filesystem aren''t cached into local shared memory, they''re just mapped across. This lets an admin build explicit shared systems, does shared ro-memory nicely, etc. Might be worth looking into implementing similar in Xen. Apologies if this was already discussed/noticed, -m On Tue, Nov 02, 2004 at 09:33:00PM +0000, Ian Pratt wrote:> > > What about having a special variant of the pte-update operation, which > > takes as input a pointer to a pte-entry, and will change that entry into > > a read-only mapping of the same page contents, though perhaps with a > > different frame number in the pte. > > I''m still not convinced we wouldn''t get most of the benefit by > just sharing pages read from the block/file system, and hence > avoiding all the hashing. > > Collecting this data is actually very easy -- its just a case of > coming up with a few "realistic usage scenarios" in which to > collect the data. We''d be happy to work with someone to do > this... > > Ian > > > ------------------------------------------------------- > This SF.Net email is sponsored by: > Sybase ASE Linux Express Edition - download now for FREE > LinuxWorld Reader''s Choice Award Winner for best database on Linux. > http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xen-devel------------------------------------------------------- This SF.Net email is sponsored by: Sybase ASE Linux Express Edition - download now for FREE LinuxWorld Reader''s Choice Award Winner for best database on Linux. http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel