Hi folks, A previous thread discussed complications around DomU''s sharing memory pages with each other: http://lists.xensource.com/archives/html/xen-devel/2005-12/msg00499.html To summarize, DomU''s get into trouble, e.g. unable to shutdown, unless the remote DomU''s play nice. Since DomU''s do not trust each other, that is problematic. I''d like to discuss how to clean away this dependency. Here''s one idea. The goal is to robustly decouple the sharing and remote domains. Grant tables add a new GTF_safe flag, settable by the sharing DomU. In order to map a GTF_safe page, a remote domain must provide a page of its own, which I''ll call an "under page". Xen holds the under-page on behalf of the remote DomU and maps the shared page into the remote DomU''s machine. At any time, the sharing DomU can unshare the page, crash, etc, which ends ALL foreign access to that page, not just new mappings. For each remote domain that still maps the unshared page, Xen maps the remote''s under-page in place of the unshared page. The remote domain can unmap at any time and recover its under-page. The purpose of the under-page is to plug the memory hole in the remote DomU created by a surprise unsharing. A nervous remote DomU could check that a share is GTF_safe before proceeding to map the page. Good, bad or ugly? -steve _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 13 Jan 2006, at 19:07, King, Steven R wrote:> The purpose of the under-page is to plug the memory hole in the remote > DomU created by a surprise unsharing. A nervous remote DomU could > check that a share is GTF_safe before proceeding to map the page. > > Good, bad or ugly?What''s wrong with the current scheme (sharing domU sticks around as a zombie until foreign mappings disappear)? This needn''t stop control tools from restarting the domain (they can see that it has shut down, for example, and is simply awaiting reaping when its refcnt falls to zero). It''s arguable the zombies needn''t even be kept on the domain list, so they become invisible to the control tools (since they''re really just a resource container for foreign page mappings). OTOH hiding things from control tools is probably a bad idea. :-) -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
An ideal solution to this problem would be to keep a separate pool of shared memory that neither domain owned. That removes any concerns about ownership. Regards, Anthony Liguori King, Steven R wrote:> Hi folks, > A previous thread discussed complications around DomU''s sharing memory > pages with each other: > http://lists.xensource.com/archives/html/xen-devel/2005-12/msg00499.html > > To summarize, DomU''s get into trouble, e.g. unable to shutdown, > unless the remote DomU''s play nice. Since DomU''s do not trust each > other, that is problematic. I''d like to discuss how to clean away > this dependency. > > Here''s one idea. The goal is to robustly decouple the sharing and > remote domains. > > Grant tables add a new GTF_safe flag, settable by the sharing DomU. > In order to map a GTF_safe page, a remote domain must provide a page > of its own, which I''ll call an "under page". > Xen holds the under-page on behalf of the remote DomU and maps > the shared page into the remote DomU''s machine. > At any time, the sharing DomU can unshare the page, crash, etc, > which ends ALL foreign access to that page, not just new mappings. > For each remote domain that still maps the unshared page, Xen maps the > remote''s under-page in place of the unshared page. > The remote domain can unmap at any time and recover its under-page. > > The purpose of the under-page is to plug the memory hole in the remote > DomU created by a surprise unsharing. A nervous remote DomU could > check that a share is GTF_safe before proceeding to map the page. > > Good, bad or ugly? > -steve > > > > >------------------------------------------------------------------------ > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Anthony -- Can you explain why this is ideal? I prefer that sharers and mappers have their own skin the game--that way, Xen doesn''t have to manage a pool and nobody has to worry about the pool being depleted. -----Original Message----- From: Anthony Liguori [mailto:aliguori@us.ibm.com] Sent: Friday, January 13, 2006 11:23 AM To: King, Steven R Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] yanked share, round 2 An ideal solution to this problem would be to keep a separate pool of shared memory that neither domain owned. That removes any concerns about ownership. Regards, Anthony Liguori King, Steven R wrote:> Hi folks, > A previous thread discussed complications around DomU''s sharing memory> pages with each other: > http://lists.xensource.com/archives/html/xen-devel/2005-12/msg00499.ht > ml > > To summarize, DomU''s get into trouble, e.g. unable to shutdown, unless> the remote DomU''s play nice. Since DomU''s do not trust each other, > that is problematic. I''d like to discuss how to clean away this > dependency. > > Here''s one idea. The goal is to robustly decouple the sharing and > remote domains. > > Grant tables add a new GTF_safe flag, settable by the sharing DomU. > In order to map a GTF_safe page, a remote domain must provide a page > of its own, which I''ll call an "under page". > Xen holds the under-page on behalf of the remote DomU and maps the > shared page into the remote DomU''s machine. > At any time, the sharing DomU can unshare the page, crash, etc, which > ends ALL foreign access to that page, not just new mappings. > For each remote domain that still maps the unshared page, Xen maps the> remote''s under-page in place of the unshared page. > The remote domain can unmap at any time and recover its under-page. > > The purpose of the under-page is to plug the memory hole in the remote> DomU created by a surprise unsharing. A nervous remote DomU could > check that a share is GTF_safe before proceeding to map the page. > > Good, bad or ugly? > -steve > > > > >----------------------------------------------------------------------- >- > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
You avoid zombies and either side can break the sharing without causing harm to the other side. Domains restarting are transparent to either end (the restarting domain just reattachs and keeps going). It avoids the general ownership problem altogether. Regards, Anthony Liguori King, Steven R wrote:>Hi Anthony -- Can you explain why this is ideal? I prefer that sharers >and mappers have their own skin the game--that way, Xen doesn''t have to >manage a pool and nobody has to worry about the pool being depleted. > >-----Original Message----- >From: Anthony Liguori [mailto:aliguori@us.ibm.com] >Sent: Friday, January 13, 2006 11:23 AM >To: King, Steven R >Cc: xen-devel@lists.xensource.com >Subject: Re: [Xen-devel] yanked share, round 2 > >An ideal solution to this problem would be to keep a separate pool of >shared memory that neither domain owned. That removes any concerns >about ownership. > >Regards, > >Anthony Liguori > >King, Steven R wrote: > > > >>Hi folks, >>A previous thread discussed complications around DomU''s sharing memory >> >> > > > >>pages with each other: >>http://lists.xensource.com/archives/html/xen-devel/2005-12/msg00499.ht >>ml >> >>To summarize, DomU''s get into trouble, e.g. unable to shutdown, unless >> >> > > > >>the remote DomU''s play nice. Since DomU''s do not trust each other, >>that is problematic. I''d like to discuss how to clean away this >>dependency. >> >>Here''s one idea. The goal is to robustly decouple the sharing and >>remote domains. >> >>Grant tables add a new GTF_safe flag, settable by the sharing DomU. >>In order to map a GTF_safe page, a remote domain must provide a page >>of its own, which I''ll call an "under page". >>Xen holds the under-page on behalf of the remote DomU and maps the >>shared page into the remote DomU''s machine. >>At any time, the sharing DomU can unshare the page, crash, etc, which >>ends ALL foreign access to that page, not just new mappings. >>For each remote domain that still maps the unshared page, Xen maps the >> >> > > > >>remote''s under-page in place of the unshared page. >>The remote domain can unmap at any time and recover its under-page. >> >>The purpose of the under-page is to plug the memory hole in the remote >> >> > > > >>DomU created by a surprise unsharing. A nervous remote DomU could >>check that a share is GTF_safe before proceeding to map the page. >> >>Good, bad or ugly? >>-steve >> >> >> >> >>----------------------------------------------------------------------- >>- >> >>_______________________________________________ >>Xen-devel mailing list >>Xen-devel@lists.xensource.com >>http://lists.xensource.com/xen-devel >> >> >> >> > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Keir, I''m not familiar enough with zombie anatomy to help there, so let me try this reasoning: today''s Xen architecture cannot promise that a shared page can ever be returned to normal, non-shared service. Thus, any DomU that routinely creates shared pages *must* eventually run out of memory. Of course if all DomU''s try to play nice, it would take a long while. We still face a snag in which Xen allows DomU bugs, DomU crashes and DomU evil to accumulate over time. By analogy to the hardware world, a PCI device that could not promise to let go of pages would be unacceptable. -steve -----Original Message----- From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] Sent: Friday, January 13, 2006 11:21 AM To: King, Steven R Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] yanked share, round 2 On 13 Jan 2006, at 19:07, King, Steven R wrote:> The purpose of the under-page is to plug the memory hole in the remote > DomU created by a surprise unsharing. A nervous remote DomU could > check that a share is GTF_safe before proceeding to map the page. > > Good, bad or ugly?What''s wrong with the current scheme (sharing domU sticks around as a zombie until foreign mappings disappear)? This needn''t stop control tools from restarting the domain (they can see that it has shut down, for example, and is simply awaiting reaping when its refcnt falls to zero). It''s arguable the zombies needn''t even be kept on the domain list, so they become invisible to the control tools (since they''re really just a resource container for foreign page mappings). OTOH hiding things from control tools is probably a bad idea. :-) -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 13 Jan 2006, at 19:55, King, Steven R wrote:> Hi Keir, I''m not familiar enough with zombie anatomy to help there, so > let me try this reasoning: today''s Xen architecture cannot promise > that a shared page can ever be returned to normal, non-shared service. > Thus, any DomU that routinely creates shared pages *must* eventually > run out of memory. Of course if all DomU''s try to play nice, it would > take a long while. We still face a snag in which Xen allows DomU > bugs, DomU crashes and DomU evil to accumulate over time. By analogy > to the hardware world, a PCI device that could not promise to let go > of pages would be unacceptable.Just as a PCI device can be reset, we can kill an offensive device driver or other service domain and restart it. I think these problems need addressing in the tools / control plane, not with extra mechanism in Xen. At the end of the day, the memory backing these bogus/buggy mappings has to come from somewhere. Even if you add mechanism such that the mapping domain is made accountable, what should its clients do when it runs out of memory and finally hits the brick wall? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 13 Jan 2006, at 20:09, Keir wrote:>Just as a PCI device can be reset, we can kill an offensive device >driver or other service domain and restart it. I think these problems >need addressing in the tools / control plane, not with extra mechanismin Xen.>[snip]Are you suggesting that operator intervention is the durable solution? Surely, that doesn''t wash. ;^)>Even if you add mechanism such that the mapping domain is >made accountable, what should its clients do when it runs >out of memory and finally hits the brick wall?Why should the mapping domain be in special danger of running out of memory? It is ''accountable'' 1 for 1 with the number of pages it maps. If a surprise unshare occurs, the mapping domain can unmap and robustly recover the associated under-page. Without advocating any particular approach, the architectural goal is that either DomU can independently decouple itself from the other, something not possible now. -steve -----Original Message----- From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] Sent: Friday, January 13, 2006 12:09 PM To: King, Steven R Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] yanked share, round 2 On 13 Jan 2006, at 19:55, King, Steven R wrote:> Hi Keir, I''m not familiar enough with zombie anatomy to help there, so> let me try this reasoning: today''s Xen architecture cannot promise > that a shared page can ever be returned to normal, non-shared service. > Thus, any DomU that routinely creates shared pages *must* eventually > run out of memory. Of course if all DomU''s try to play nice, it would> take a long while. We still face a snag in which Xen allows DomU > bugs, DomU crashes and DomU evil to accumulate over time. By analogy > to the hardware world, a PCI device that could not promise to let go > of pages would be unacceptable.Just as a PCI device can be reset, we can kill an offensive device driver or other service domain and restart it. I think these problems need addressing in the tools / control plane, not with extra mechanism in Xen. At the end of the day, the memory backing these bogus/buggy mappings has to come from somewhere. Even if you add mechanism such that the mapping domain is made accountable, what should its clients do when it runs out of memory and finally hits the brick wall? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I would like hear your ideas for who manages the pool and how the pool avoids becoming depleted. In addition to avoiding the ownership problem, I see another nice advantage: The third party (Xen? a DomP?) can hand up to the DomU''s a nice tidy key value representing the shared pages, which is very similar to the way SysV IPC memory sharing works. -steve -----Original Message----- From: Anthony Liguori [mailto:aliguori@us.ibm.com] Sent: Friday, January 13, 2006 11:35 AM To: King, Steven R Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] yanked share, round 2 You avoid zombies and either side can break the sharing without causing harm to the other side. Domains restarting are transparent to either end (the restarting domain just reattachs and keeps going). It avoids the general ownership problem altogether. Regards, Anthony Liguori King, Steven R wrote:>Hi Anthony -- Can you explain why this is ideal? I prefer that sharers>and mappers have their own skin the game--that way, Xen doesn''t have to>manage a pool and nobody has to worry about the pool being depleted. > >-----Original Message----- >From: Anthony Liguori [mailto:aliguori@us.ibm.com] >Sent: Friday, January 13, 2006 11:23 AM >To: King, Steven R >Cc: xen-devel@lists.xensource.com >Subject: Re: [Xen-devel] yanked share, round 2 > >An ideal solution to this problem would be to keep a separate pool of >shared memory that neither domain owned. That removes any concerns >about ownership. > >Regards, > >Anthony Liguori > >King, Steven R wrote: > > > >>Hi folks, >>A previous thread discussed complications around DomU''s sharing memory >> >> > > > >>pages with each other: >>http://lists.xensource.com/archives/html/xen-devel/2005-12/msg00499.ht >>ml >> >>To summarize, DomU''s get into trouble, e.g. unable to shutdown, unless >> >> > > > >>the remote DomU''s play nice. Since DomU''s do not trust each other, >>that is problematic. I''d like to discuss how to clean away this >>dependency. >> >>Here''s one idea. The goal is to robustly decouple the sharing and >>remote domains. >> >>Grant tables add a new GTF_safe flag, settable by the sharing DomU. >>In order to map a GTF_safe page, a remote domain must provide a page >>of its own, which I''ll call an "under page". >>Xen holds the under-page on behalf of the remote DomU and maps the >>shared page into the remote DomU''s machine. >>At any time, the sharing DomU can unshare the page, crash, etc, which >>ends ALL foreign access to that page, not just new mappings. >>For each remote domain that still maps the unshared page, Xen maps the >> >> > > > >>remote''s under-page in place of the unshared page. >>The remote domain can unmap at any time and recover its under-page. >> >>The purpose of the under-page is to plug the memory hole in the remote >> >> > > > >>DomU created by a surprise unsharing. A nervous remote DomU could >>check that a share is GTF_safe before proceeding to map the page. >> >>Good, bad or ugly? >>-steve >> >> >> >> >>---------------------------------------------------------------------- >>- >>- >> >>_______________________________________________ >>Xen-devel mailing list >>Xen-devel@lists.xensource.com >>http://lists.xensource.com/xen-devel >> >> >> >> > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
King, Steven R wrote:>I would like hear your ideas for who manages the pool and how the pool >avoids becoming depleted. > >I won''t take credit for this as it is Rusty''s idea actually :-) Depletion can be handled by setting a maximum amount of shared memory per-domain (2MB for instance). Then as long as there is enough free memory to satisfy the per-domain sharing requirement, you''re fine. The memory can be allocated straight from the xen heap and referenced counted such that it is returned back to the heap when the last user stops sharing it. The 2MB limit would be somewhat virtual since the same page would be factored into every domain''s actual limit who had a reference to the page. This is my understand at least, perhaps Rusty can clarify with what he was thinking.>In addition to avoiding the ownership problem, I see another nice >advantage: >The third party (Xen? a DomP?) can hand up to the DomU''s a nice tidy key >value representing the shared pages, which is very similar to the way >SysV IPC memory sharing works. > >Precisely :-) Regards, Anthony Liguori>-steve > >-----Original Message----- >From: Anthony Liguori [mailto:aliguori@us.ibm.com] >Sent: Friday, January 13, 2006 11:35 AM >To: King, Steven R >Cc: xen-devel@lists.xensource.com >Subject: Re: [Xen-devel] yanked share, round 2 > >You avoid zombies and either side can break the sharing without causing >harm to the other side. Domains restarting are transparent to either >end (the restarting domain just reattachs and keeps going). It avoids >the general ownership problem altogether. > >Regards, > >Anthony Liguori > >King, Steven R wrote: > > > >>Hi Anthony -- Can you explain why this is ideal? I prefer that sharers >> >> > > > >>and mappers have their own skin the game--that way, Xen doesn''t have to >> >> > > > >>manage a pool and nobody has to worry about the pool being depleted. >> >>-----Original Message----- >>From: Anthony Liguori [mailto:aliguori@us.ibm.com] >>Sent: Friday, January 13, 2006 11:23 AM >>To: King, Steven R >>Cc: xen-devel@lists.xensource.com >>Subject: Re: [Xen-devel] yanked share, round 2 >> >>An ideal solution to this problem would be to keep a separate pool of >>shared memory that neither domain owned. That removes any concerns >>about ownership. >> >>Regards, >> >>Anthony Liguori >> >>King, Steven R wrote: >> >> >> >> >> >>>Hi folks, >>>A previous thread discussed complications around DomU''s sharing memory >>> >>> >>> >>> >> >> >> >> >>>pages with each other: >>>http://lists.xensource.com/archives/html/xen-devel/2005-12/msg00499.ht >>>ml >>> >>>To summarize, DomU''s get into trouble, e.g. unable to shutdown, unless >>> >>> >>> >>> >> >> >> >> >>>the remote DomU''s play nice. Since DomU''s do not trust each other, >>>that is problematic. I''d like to discuss how to clean away this >>>dependency. >>> >>>Here''s one idea. The goal is to robustly decouple the sharing and >>>remote domains. >>> >>>Grant tables add a new GTF_safe flag, settable by the sharing DomU. >>>In order to map a GTF_safe page, a remote domain must provide a page >>>of its own, which I''ll call an "under page". >>>Xen holds the under-page on behalf of the remote DomU and maps the >>>shared page into the remote DomU''s machine. >>>At any time, the sharing DomU can unshare the page, crash, etc, which >>>ends ALL foreign access to that page, not just new mappings. >>>For each remote domain that still maps the unshared page, Xen maps the >>> >>> >>> >>> >> >> >> >> >>>remote''s under-page in place of the unshared page. >>>The remote domain can unmap at any time and recover its under-page. >>> >>>The purpose of the under-page is to plug the memory hole in the remote >>> >>> >>> >>> >> >> >> >> >>>DomU created by a surprise unsharing. A nervous remote DomU could >>>check that a share is GTF_safe before proceeding to map the page. >>> >>>Good, bad or ugly? >>>-steve >>> >>> >>> >>> >>>---------------------------------------------------------------------- >>>- >>>- >>> >>>_______________________________________________ >>>Xen-devel mailing list >>>Xen-devel@lists.xensource.com >>>http://lists.xensource.com/xen-devel >>> >>> >>> >>> >>> >>> >> >> >> >> > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
How about combining both ideas: In order to map N pages from Xen''s share pool, a DomU must provide N pages back to the pool. When a DomU unmaps (or crashes), it gets N pages back. We get these features: 1) A domU can never cause the Xen share pool to shrink. 2) The number of pages mappable by a DomU is bounded only by the DomU itself. 3) No page ownership problems. 4) We can have a nice share key ala SysV IPC semantics. 5) When no shares exist, the memory pool consumes no pages. 6) We leave Xen''s heap alone. The first benefit is critical since DomU''s are untrusted. A downside is that a platform with many DomU mappings will have many idle pages sitting in the pool. Given all the benefits above, a very reasonable price to pay. -steve -----Original Message----- From: Anthony Liguori [mailto:aliguori@us.ibm.com] Sent: Friday, January 13, 2006 1:32 PM To: King, Steven R Cc: Rusty Russell; xen-devel Subject: Re: [Xen-devel] yanked share, round 2 King, Steven R wrote:>I would like hear your ideas for who manages the pool and how the pool >avoids becoming depleted. > >I won''t take credit for this as it is Rusty''s idea actually :-) Depletion can be handled by setting a maximum amount of shared memory per-domain (2MB for instance). Then as long as there is enough free memory to satisfy the per-domain sharing requirement, you''re fine. The memory can be allocated straight from the xen heap and referenced counted such that it is returned back to the heap when the last user stops sharing it. The 2MB limit would be somewhat virtual since the same page would be factored into every domain''s actual limit who had a reference to the page. This is my understand at least, perhaps Rusty can clarify with what he was thinking.>In addition to avoiding the ownership problem, I see another nice >advantage: >The third party (Xen? a DomP?) can hand up to the DomU''s a nice tidy >key value representing the shared pages, which is very similar to the >way SysV IPC memory sharing works. > >Precisely :-) Regards, Anthony Liguori>-steve > >-----Original Message----- >From: Anthony Liguori [mailto:aliguori@us.ibm.com] >Sent: Friday, January 13, 2006 11:35 AM >To: King, Steven R >Cc: xen-devel@lists.xensource.com >Subject: Re: [Xen-devel] yanked share, round 2 > >You avoid zombies and either side can break the sharing without causing>harm to the other side. Domains restarting are transparent to either >end (the restarting domain just reattachs and keeps going). It avoids >the general ownership problem altogether. > >Regards, > >Anthony Liguori > >King, Steven R wrote: > > > >>Hi Anthony -- Can you explain why this is ideal? I prefer that >>sharers >> >> > > > >>and mappers have their own skin the game--that way, Xen doesn''t have >>to >> >> > > > >>manage a pool and nobody has to worry about the pool being depleted. >> >>-----Original Message----- >>From: Anthony Liguori [mailto:aliguori@us.ibm.com] >>Sent: Friday, January 13, 2006 11:23 AM >>To: King, Steven R >>Cc: xen-devel@lists.xensource.com >>Subject: Re: [Xen-devel] yanked share, round 2 >> >>An ideal solution to this problem would be to keep a separate pool of >>shared memory that neither domain owned. That removes any concerns >>about ownership. >> >>Regards, >> >>Anthony Liguori >> >>King, Steven R wrote: >> >> >> >> >> >>>Hi folks, >>>A previous thread discussed complications around DomU''s sharing >>>memory >>> >>> >>> >>> >> >> >> >> >>>pages with each other: >>>http://lists.xensource.com/archives/html/xen-devel/2005-12/msg00499.h >>>t >>>ml >>> >>>To summarize, DomU''s get into trouble, e.g. unable to shutdown, >>>unless >>> >>> >>> >>> >> >> >> >> >>>the remote DomU''s play nice. Since DomU''s do not trust each other, >>>that is problematic. I''d like to discuss how to clean away this >>>dependency. >>> >>>Here''s one idea. The goal is to robustly decouple the sharing and >>>remote domains. >>> >>>Grant tables add a new GTF_safe flag, settable by the sharing DomU. >>>In order to map a GTF_safe page, a remote domain must provide a page >>>of its own, which I''ll call an "under page". >>>Xen holds the under-page on behalf of the remote DomU and maps the >>>shared page into the remote DomU''s machine. >>>At any time, the sharing DomU can unshare the page, crash, etc, which>>>ends ALL foreign access to that page, not just new mappings. >>>For each remote domain that still maps the unshared page, Xen maps >>>the >>> >>> >>> >>> >> >> >> >> >>>remote''s under-page in place of the unshared page. >>>The remote domain can unmap at any time and recover its under-page. >>> >>>The purpose of the under-page is to plug the memory hole in the >>>remote >>> >>> >>> >>> >> >> >> >> >>>DomU created by a surprise unsharing. A nervous remote DomU could >>>check that a share is GTF_safe before proceeding to map the page. >>> >>>Good, bad or ugly? >>>-steve >>> >>> >>> >>> >>>--------------------------------------------------------------------- >>>- >>>- >>>- >>> >>>_______________________________________________ >>>Xen-devel mailing list >>>Xen-devel@lists.xensource.com >>>http://lists.xensource.com/xen-devel >>> >>> >>> >>> >>> >>> >> >> >> >> > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 13 Jan 2006, at 23:02, King, Steven R wrote:> We get these features: > 1) A domU can never cause the Xen share pool to shrink. > 2) The number of pages mappable by a DomU is bounded only by the DomU > itself. > 3) No page ownership problems. > 4) We can have a nice share key ala SysV IPC semantics. > 5) When no shares exist, the memory pool consumes no pages. > 6) We leave Xen''s heap alone. > > The first benefit is critical since DomU''s are untrusted. A downside > is > that a platform with many DomU mappings will have many idle pages > sitting in the pool. Given all the benefits above, a very reasonable > price to pay.This is similar to ideas people have had for shared buffer cache without memory overcommitment. Seems to me that the Xen-level mechanism for doing the sharing could easily be identical, and the differences in interface, access control and other policy could be hidden in a share manager in a suitably-privileged domain. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 2006-01-13 at 15:02 -0800, King, Steven R wrote:> How about combining both ideas: In order to map N pages from Xen''s > share pool, a DomU must provide N pages back to the pool. When a DomU > unmaps (or crashes), it gets N pages back.OK, early last year I hacked up a prototype implementation, which I''m now bringing up-to-date with the modern Xen code, and some new simplifications I''m toying with. It''s certainly a simple model to never have domains share their own pages, and the hypervisor provide a DMA-like mechanism between domains to allow bulk transfer. This would then page-swap if appropriate. It makes drivers nice and simple, and avoids problems with starvation et. al. Earlier results suggested it is competitive on benchmarks, too, but it needs real testing under various loads. Noone here should be surprised though, that I regard the simplicity of the scheme as the most important factor. Cheers! Rusty. -- ccontrol: http://ozlabs.org/~rusty/ccontrol _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> The third party (Xen? a DomP?) can hand up to the DomU''s a nice tidy key > value representing the shared pages, which is very similar to the way > SysV IPC memory sharing works.This is starting to sound rather like Saved Segments on IBM zSeries to me... Also sounds like named shared regions could be managed by dom0, with control plane communication (naming, grant refs, etc) occurring through XenStore - i.e. using a share back/front driver. Cheers, Mark> -steve > > -----Original Message----- > From: Anthony Liguori [mailto:aliguori@us.ibm.com] > Sent: Friday, January 13, 2006 11:35 AM > To: King, Steven R > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] yanked share, round 2 > > You avoid zombies and either side can break the sharing without causing > harm to the other side. Domains restarting are transparent to either > end (the restarting domain just reattachs and keeps going). It avoids > the general ownership problem altogether. > > Regards, > > Anthony Liguori > > King, Steven R wrote: > >Hi Anthony -- Can you explain why this is ideal? I prefer that sharers > > > >and mappers have their own skin the game--that way, Xen doesn''t have to > > > >manage a pool and nobody has to worry about the pool being depleted. > > > >-----Original Message----- > >From: Anthony Liguori [mailto:aliguori@us.ibm.com] > >Sent: Friday, January 13, 2006 11:23 AM > >To: King, Steven R > >Cc: xen-devel@lists.xensource.com > >Subject: Re: [Xen-devel] yanked share, round 2 > > > >An ideal solution to this problem would be to keep a separate pool of > >shared memory that neither domain owned. That removes any concerns > >about ownership. > > > >Regards, > > > >Anthony Liguori > > > >King, Steven R wrote: > >>Hi folks, > >>A previous thread discussed complications around DomU''s sharing memory > >> > >> > >> > >> > >> > >>pages with each other: > >>http://lists.xensource.com/archives/html/xen-devel/2005-12/msg00499.ht > >>ml > >> > >>To summarize, DomU''s get into trouble, e.g. unable to shutdown, unless > >> > >> > >> > >> > >> > >>the remote DomU''s play nice. Since DomU''s do not trust each other, > >>that is problematic. I''d like to discuss how to clean away this > >>dependency. > >> > >>Here''s one idea. The goal is to robustly decouple the sharing and > >>remote domains. > >> > >>Grant tables add a new GTF_safe flag, settable by the sharing DomU. > >>In order to map a GTF_safe page, a remote domain must provide a page > >>of its own, which I''ll call an "under page". > >>Xen holds the under-page on behalf of the remote DomU and maps the > >>shared page into the remote DomU''s machine. > >>At any time, the sharing DomU can unshare the page, crash, etc, which > >>ends ALL foreign access to that page, not just new mappings. > >>For each remote domain that still maps the unshared page, Xen maps the > >> > >> > >> > >> > >> > >>remote''s under-page in place of the unshared page. > >>The remote domain can unmap at any time and recover its under-page. > >> > >>The purpose of the under-page is to plug the memory hole in the remote > >> > >> > >> > >> > >> > >>DomU created by a surprise unsharing. A nervous remote DomU could > >>check that a share is GTF_safe before proceeding to map the page. > >> > >>Good, bad or ugly? > >>-steve > >> > >> > >> > >> > >>---------------------------------------------------------------------- > >>- > >>- > >> > >>_______________________________________________ > >>Xen-devel mailing list > >>Xen-devel@lists.xensource.com > >>http://lists.xensource.com/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel-- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> This is similar to ideas people have had for shared buffer cache > without memory overcommitment.Quite similar to the mechanisms I''m incorporating into XenFS, for one thing.> Seems to me that the Xen-level mechanism > for doing the sharing could easily be identical, and the differences in > interface, access control and other policy could be hidden in a share > manager in a suitably-privileged domain.Agreed, doesn''t seem like it needs to be an in-Xen service. Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Hi Keir, I''m not familiar enough with zombie anatomy to help there, so let > me try this reasoning: today''s Xen architecture cannot promise that a > shared page can ever be returned to normal, non-shared service. Thus, any > DomU that routinely creates shared pages *must* eventually run out of > memory.Eveni if it could reboot, causing the control tools to allocate it new memory... the more worrying problem is that a domain that does *not* return shared pages could eventually end up holding all the memory in the machine, with it all being credited to other (zombie) domains.> Of course if all DomU''s try to play nice, it would take a long > while. We still face a snag in which Xen allows DomU bugs, DomU crashes > and DomU evil to accumulate over time. By analogy to the hardware world, a > PCI device that could not promise to let go of pages would be unacceptable.A centralised store (managed by a trusted entity) would be rather good for domUs that really don''t trust each other. It''d be worth having a driver that could set these up - presumably with shares advertised using XenStore. Cheers, Mark> -steve > > > > -----Original Message----- > From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] > Sent: Friday, January 13, 2006 11:21 AM > To: King, Steven R > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] yanked share, round 2 > > > On 13 Jan 2006, at 19:07, King, Steven R wrote: > > > > The purpose of the under-page is to plug the memory hole in the remote > > DomU created by a surprise unsharing. A nervous remote DomU could > > check that a share is GTF_safe before proceeding to map the page. > > > > Good, bad or ugly? > > What''s wrong with the current scheme (sharing domU sticks around as a > zombie until foreign mappings disappear)? This needn''t stop control tools > from restarting the domain (they can see that it has shut down, for > example, and is simply awaiting reaping when its refcnt falls to zero). > > It''s arguable the zombies needn''t even be kept on the domain list, so they > become invisible to the control tools (since they''re really just a resource > container for foreign page mappings). OTOH hiding things from control tools > is probably a bad idea. :-) > > -- Keir > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel-- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel