Hi Folks, Let''s imagine we have two domains, DomU_A and DomU_B. Being DomU''s, the two domains don''t trust each other, but still desire to safely setup a shared memory page. The scenario is the following: DomU_A acquires a grant reference and shares a page to DomU_B. DomU_B gets the grant reference passed from A and maps the shared page. DomU_A then, due to bug or badness, unshares the page without telling DomU_B. DomU_B faults on the next access to the mapped but unshared page. Since one DomU causing a fault in another DomU is unacceptable, the two domains cannot share memory. Have I missed something? I would much appreciate being set straight on this. Thanks, -steve _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 13 Dec 2005, at 23:05, King, Steven R wrote:> Since one DomU causing a fault in another DomU is unacceptable, the two > domains cannot share memory. Have I missed something? I would much > appreciate being set straight on this.Yes you have. There is no forced revocation of mapped grant references. The only supported revocation model is ''big hammer'' (== destroy the domain who is mapping the page). -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Unsharing the page doesn''t forcibly revoke other domain''s mappings to it, it just removes their right to create mappings. DomU_A in your scenario wouldn''t be able to cause the fault in DomU_B, the mapping to the page would be there until domU_B had finished with it. The moral of the story is: don''t grant pages to people who might not give them back ;-) Cheers. Mark On Tuesday 13 December 2005 23:05, King, Steven R wrote:> Hi Folks, > > Let''s imagine we have two domains, DomU_A and DomU_B. Being DomU''s, the > two domains don''t trust each other, but still desire to safely setup a > shared memory page. > > The scenario is the following: > > DomU_A acquires a grant reference and shares a page to DomU_B. > DomU_B gets the grant reference passed from A and maps the shared page. > DomU_A then, due to bug or badness, unshares the page without telling > DomU_B. > DomU_B faults on the next access to the mapped but unshared page. > > Since one DomU causing a fault in another DomU is unacceptable, the two > domains cannot share memory. Have I missed something? I would much > appreciate being set straight on this. > > Thanks, > -steve > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 12/14/05, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:> Unsharing the page doesn''t forcibly revoke other domain''s mappings to it, it > just removes their right to create mappings. DomU_A in your scenario > wouldn''t be able to cause the fault in DomU_B, the mapping to the page would > be there until domU_B had finished with it. >Could you tell me what happen if the DomU_A crash while DomU_B still accesses the memory it is granted? And moreover, how can DomU_A knows that his friend has just "died"? Many thanks. Hieu _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Could you tell me what happen if the DomU_A crash while DomU_B still > accesses the memory it is granted? And moreover, how can DomU_A knows > that his friend has just "died"?AFAIK, A will be prevented from being fully destroyed until B drops the reference to that page of memory. The page will be around as long as B wants it. Cheers, Mark _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>From: Mark Williamson >Sent: 2005年12月14日 22:30 > >> Could you tell me what happen if the DomU_A crash while DomU_B still >> accesses the memory it is granted? And moreover, how can DomU_A knows >> that his friend has just "died"? > >AFAIK, A will be prevented from being fully destroyed until B drops the >reference to that page of memory. The page will be around as long as B wants >it. > >Cheers, >Mark >How about B is waiting for A''s notification to end reference, but A crashed before sending out notification? One immediate example is the shared ring buf between backend and frontend. Backend may not access the shared ring buf when A is crashed. But that doesn''t mean backend won''t access that address later since that virtual address is legally allocated from linux buddy pool. We need provide a way to notify reference side something going abnormally, and let reference side to drop reference and release local resource. One possible way is to register a grant call back for each driver. When Xen detects A crashed, xen notifies registered callback. For example, backend can register a callback which check whether any reference on-going. If yes, waiting for those reference done. Finally release all references to grant entries of crashed domain and also release local resource back to linux and exit the driver. After all callbacks are done, Xen then free that machine page. Thanks Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> >> Could you tell me what happen if the DomU_A crash while DomU_B still > >> accesses the memory it is granted? And moreover, how can DomU_A knows > >> that his friend has just "died"? > > > >AFAIK, A will be prevented from being fully destroyed until B drops the > >reference to that page of memory. The page will be around as long as B > > wants it. > > > >Cheers, > >Mark > > How about B is waiting for A''s notification to end reference, but A crashed > before sending out notification? One immediate example is the shared ring > buf between backend and frontend. Backend may not access the shared ring > buf when A is crashed. But that doesn''t mean backend won''t access that > address later since that virtual address is legally allocated from linux > buddy pool. We need provide a way to notify reference side something going > abnormally, and let reference side to drop reference and release local > resource.So, if the frontend domain crashes but the backend driver is still accessing the comms ring? It won''t actually break things if the backend accesses the comms ring for a crashed domain, it just won''t be able to do sensible IO requests anymore. In the case you describe, the virtual device in the backend would get destroyed, since the device would disappear out of Xenstore when the crashed domain is destroyed. This will cause the backend to unmap the granted page, which''ll then get returned to Xen (allowing the frontend domain to be fully destroyed).> One possible way is to register a grant call back for each driver. When Xen > detects A crashed, xen notifies registered callback. For example, backend > can register a callback which check whether any reference on-going. If yes, > waiting for those reference done. Finally release all references to grant > entries of crashed domain and also release local resource back to linux and > exit the driver. After all callbacks are done, Xen then free that machine > page.I think you should be able to achieve most of what you want by co-ordinating access to the share using Xenstore: you''ll need to use the store to set up the location of the shared memory anyhow, so you might as well use it to be notified of when the other domain goes away? Does that sound about right? Cheers, Mark _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>From: Mark Williamson [mailto:mark.williamson@cl.cam.ac.uk] >Sent: 2005年12月15日 0:57 > >> >> Could you tell me what happen if the DomU_A crash while DomU_B still >> >> accesses the memory it is granted? And moreover, how can DomU_A knows >> >> that his friend has just "died"? >> > >> >AFAIK, A will be prevented from being fully destroyed until B drops the >> >reference to that page of memory. The page will be around as long as B >> > wants it. >> > >> >Cheers, >> >Mark >> >> How about B is waiting for A''s notification to end reference, but A crashed >> before sending out notification? One immediate example is the shared ring >> buf between backend and frontend. Backend may not access the shared ring >> buf when A is crashed. But that doesn''t mean backend won''t access that >> address later since that virtual address is legally allocated from linux >> buddy pool. We need provide a way to notify reference side something going >> abnormally, and let reference side to drop reference and release local >> resource. > >So, if the frontend domain crashes but the backend driver is still accessing >the comms ring? > >It won''t actually break things if the backend accesses the comms ring for a >crashed domain, it just won''t be able to do sensible IO requests anymore. In >the case you describe, the virtual device in the backend would get destroyed, >since the device would disappear out of Xenstore when the crashed domain is >destroyed. This will cause the backend to unmap the granted page, which''ll >then get returned to Xen (allowing the frontend domain to be fully >destroyed). > >> One possible way is to register a grant call back for each driver. When Xen >> detects A crashed, xen notifies registered callback. For example, backend >> can register a callback which check whether any reference on-going. If yes, >> waiting for those reference done. Finally release all references to grant >> entries of crashed domain and also release local resource back to linux and >> exit the driver. After all callbacks are done, Xen then free that machine >> page. > >I think you should be able to achieve most of what you want by co-ordinating >access to the share using Xenstore: you''ll need to use the store to set up >the location of the shared memory anyhow, so you might as well use it to be >notified of when the other domain goes away? > >Does that sound about right? > >Cheers, >MarkYes, that''s the desired way in most cases, as long as domain crash event can be notified to xenstore and then watches upon specific node can work for this purpose. Normal shutdown command issued from user can follow this process due to driver code aware to send traffic on xenbus. But abnormal crash may not go to such branch. Maybe we need a heartbeat check between xenstore and hooked domains? ;-) Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Yes, that''s the desired way in most cases, as long as domain crash event > can be notified to xenstore and then watches upon specific node can work > for this purpose. Normal shutdown command issued from user can follow > this process due to driver code aware to send traffic on xenbus. But > abnormal crash may not go to such branch. Maybe we need a heartbeat check > between xenstore and hooked domains? ;-)When a domain dies (for any reason), a VIRQ_DOM_EXC will be sent to dom0, telling it that something "exceptional" (like being shutdown, or crashing) has happened to a domain. This causes some entity (currently still Xend, AFAIK) to go check the state of all the domains - the store will get updated appropriately as a result of this. This should work even for crashes - in the current world, this mechanism also ensures it''s possible to restart domains, clean up their devices, etc if they die unexpectedly. So watching the store ought to be fine in all cases, so long as dom0 stays up (which is another problem ;-) Cheers, Mark _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
M.A. Williamson wrote:>> Yes, that''s the desired way in most cases, as long as domain crash >> event can be notified to xenstore and then watches upon specific node >> can work for this purpose. Normal shutdown command issued from user >> can follow this process due to driver code aware to send traffic on >> xenbus. But abnormal crash may not go to such branch. Maybe we need a >> heartbeat check between xenstore and hooked domains? ;-) > > > When a domain dies (for any reason), a VIRQ_DOM_EXC will be sent to > dom0, telling it that something "exceptional" (like being shutdown, or > crashing) has happened to a domain. This causes some entity (currently > still Xend, AFAIK) to go check the state of all the domains - the > store will get updated appropriately as a result of this.Actually, now the VIRQ_DOM_EXC is delivered to xenstored which results in watches on @releaseDomain to be triggered. Xend set watches on @releaseDomain and does the appropriate action.> This should work even for crashes - in the current world, this > mechanism also ensures it''s possible to restart domains, clean up > their devices, etc if they die unexpectedly.If you''re really concerned about the backend freeing it''s resources, the most reliable approach (albiet somewhat hacky) is to have the backend periodically make a getdomaininfo() call and check to see if shutdown, crashed, or dying bits are set and cleanup automatically.> So watching the store ought to be fine in all cases, so long as dom0 > stays up (which is another problem ;-)Until we can kexec() dom0 in place :-) Regards, Anthony Liguori> Cheers, > Mark > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>From: M.A.Williamson >Sent: 2005年12月15日 10:00 > >> Yes, that''s the desired way in most cases, as long as domain crash event >> can be notified to xenstore and then watches upon specific node can work >> for this purpose. Normal shutdown command issued from user can follow >> this process due to driver code aware to send traffic on xenbus. But >> abnormal crash may not go to such branch. Maybe we need a heartbeat check >> between xenstore and hooked domains? ;-) > >When a domain dies (for any reason), a VIRQ_DOM_EXC will be sent to dom0, >telling it that something "exceptional" (like being shutdown, or crashing) >has happened to a domain. This causes some entity (currently still Xend, >AFAIK) to go check the state of all the domains - the store will get >updated appropriately as a result of this. > >This should work even for crashes - in the current world, this mechanism >also ensures it''s possible to restart domains, clean up their devices, etc >if they die unexpectedly. > >So watching the store ought to be fine in all cases, so long as dom0 stays >up (which is another problem ;-) >OK, that''s clearer now. Yes, as long as xenstore keeps working, such case can always be handled. Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel