Hi, I would like to announce the first release of Kemari, an open-source virtual machine synchronization mechanism for fault tolerance. It offers a feasible approach to fault tolerance that does not require the use of specific hardware or modification of applications/OS. Kemari aims to keep VMs transparently running in times of hardware failures. It transfers the state of the primary VM to the secondary VM when the primary VM is about to send an event to devices such as storage and networks. The source code and further information are available at the following site. http://www.osrg.net/kemari/ Older slides and video are also available at, http://www.xen.org/community/projects.html In addition to Linux with PV Drivers, we tested Windows with James Harpers''s GPL PV. The video (http://www.osrg.net/kemari/download/kemari_windows.mov) demonstrates that Windows continued transparently when power failure occurred to the primary server. Kemari is now listed on Xen Product Roadmap, (http://www.xen.org/download/roadmap.html) and we would like to ask for your reviews and comments. Briefly, Kemari in VMM taps event channel, pauses the guest (not suspend), prepares for transfer, and Kemari in userland transfers the guest. On failover, Kemari on the secondary restores the guest, and the backend drivers in dom0 set up the backend rings from the state of the shared rings in the guest. Could you please review the code? Any comments are appreciated. We''re also planning to merge with Remus from UBC to propose a better solution for Xen 3.4 together. Thanks, Yoshi -- TAMURA, Yoshiaki NTT Cyber Space Labs OSS Computing Project Kernel Group E-mail: tamura.yoshiaki@lab.ntt.co.jp TEL: +81-46-859-2771 FAX: +81-46-855-1152 Address: 1-1 Hikarinooka, Yokosuka Kanagawa 239-0847 JAPAN _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi, We''re currently preparing to submit patches of Kemari. Since I didn''t give presentation at the summit last week, I would like to share the design document of Kemari, which is attached to this message. Any comments are greatly appreciated. We''ll submit patches based on this design this week. You can also get the undivided patches from the following links. http://www.osrg.net/kemari/download/kemari-v1-xen-20081120.patch http://www.osrg.net/kemari/download/kemari-v1-linux-20081120.patch If you need further information on the design or how it works, please visit http://www.osrg.net/kemari/ Thanks, Yoshi -- TAMURA, Yoshiaki NTT Cyber Space Labs OSS Computing Project Kernel Group E-mail: tamura.yoshiaki@lab.ntt.co.jp TEL: +81-46-859-2771 FAX: +81-46-855-1152 Address: 1-1 Hikarinooka, Yokosuka Kanagawa 239-0847 JAPAN _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi, We''re currently preparing to submit patches of Kemari. Since I didn''t give presentation at the summit last week, I would like to share the design document of Kemari, which is attached to this message. Any comments are greatly appreciated. We''ll submit patches based on this design this week. You can also get the undivided patches from the following links. http://www.osrg.net/kemari/download/kemari-v1-xen-20081120.patch http://www.osrg.net/kemari/download/kemari-v1-linux-20081120.patch If you need further information on the design or how it works, please visit http://www.osrg.net/kemari/ Thanks, Yoshi -- TAMURA, Yoshiaki NTT Cyber Space Labs OSS Computing Project Kernel Group E-mail: tamura.yoshiaki@lab.ntt.co.jp TEL: +81-46-859-2771 FAX: +81-46-855-1152 Address: 1-1 Hikarinooka, Yokosuka Kanagawa 239-0847 JAPAN _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 02/03/2009 13:30, "Yoshiaki Tamura" <tamura.yoshiaki@lab.ntt.co.jp> wrote:> You can also get the undivided patches from the following links. > http://www.osrg.net/kemari/download/kemari-v1-xen-20081120.patch > http://www.osrg.net/kemari/download/kemari-v1-linux-20081120.patchThe two obvious things I note are the duplicated save/restore functions, which will simply bitrot, and the additions to evtchn.c. In particular the hack to relax a validity check in evtchn_bind_interdomain() looks pretty suspicious. Apart from that there are various spurious white-space and debug-tracing changes which I assume you would clean up anyway. Those are just the things that jump out from a quick read through. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir, Thanks for your quick review. I''m glad my messages have finally reached to the list. Although the issues you pointed out here aren''t fixed yet, let me post the divided patches which are updated a little. We''ll keep up cleaning and updating the code. Yoshi Keir Fraser wrote:> On 02/03/2009 13:30, "Yoshiaki Tamura" <tamura.yoshiaki@lab.ntt.co.jp> > wrote: > >> You can also get the undivided patches from the following links. >> http://www.osrg.net/kemari/download/kemari-v1-xen-20081120.patch >> http://www.osrg.net/kemari/download/kemari-v1-linux-20081120.patch > > The two obvious things I note are the duplicated save/restore functions, > which will simply bitrot, and the additions to evtchn.c. In particular the > hack to relax a validity check in evtchn_bind_interdomain() looks pretty > suspicious. Apart from that there are various spurious white-space and > debug-tracing changes which I assume you would clean up anyway. Those are > just the things that jump out from a quick read through. > > -- Keir > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Mar 06, 2009 at 09:52:21AM +0900, Yoshiaki Tamura wrote:> Thanks for your quick review. > I''m glad my messages have finally reached to the list.I wonder how Kemari differs from Remus? regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
John Levon wrote:> On Fri, Mar 06, 2009 at 09:52:21AM +0900, Yoshiaki Tamura wrote: > >> Thanks for your quick review. >> I''m glad my messages have finally reached to the list. > > I wonder how Kemari differs from Remus? > > regards > john > > >I believe Kemari and Remus shares a lot in common. Stephen and Ian have described both Kemari and Remus on the roadmap document, and that might help for quick review. http://blog.xen.org/index.php/2008/12/08/xen-roadmap-document-2/ In short, Kemari transfers the guest VM on event-driven, while Remus transfers on specific interval. Thanks, Yoshi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sat, Mar 07, 2009 at 01:48:22AM +0900, Yoshiaki Tamura wrote:> I believe Kemari and Remus shares a lot in common. > Stephen and Ian have described both Kemari and Remus on the roadmap > document, and that might help for quick review. > > http://blog.xen.org/index.php/2008/12/08/xen-roadmap-document-2/ > > In short, Kemari transfers the guest VM on event-driven, while Remus > transfers on specific interval.Thanks. Do you have performance numbers? It sounds like your approach is intended to give lower I/O response latencies, but Remus will give better throughput with high I/O load as it doesn''t sync on every single outgoing I/O. If both projects are indeed headed for the tree, people are going to need some ideas of which to use... regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
John Levon wrote:> On Sat, Mar 07, 2009 at 01:48:22AM +0900, Yoshiaki Tamura wrote: > >> I believe Kemari and Remus shares a lot in common. >> Stephen and Ian have described both Kemari and Remus on the roadmap >> document, and that might help for quick review. >> >> http://blog.xen.org/index.php/2008/12/08/xen-roadmap-document-2/ >> >> In short, Kemari transfers the guest VM on event-driven, while Remus >> transfers on specific interval. > > Thanks. Do you have performance numbers? It sounds like your approach is > intended to give lower I/O response latencies, but Remus will give > better throughput with high I/O load as it doesn''t sync on every single > outgoing I/O.Thanks for your interest. Although the numbers are old, you can get an extended abstract and slides which I used at Xen Summit from http://www.osrg.net/kemari The patchset I sent today seems to perform a lot better, but I don''t have papers yet. Regarding the throughput, possible idea is to buffer some events to reduce the frequency, which would introduce latencies as a trade off. Although we haven''t implemented this feature yet, it may be interesting how the numbers change.> If both projects are indeed headed for the tree, people are going to > need some ideas of which to use...Well, I sent the following message when I open-sourced Kemari in November. http://lists.xensource.com/archives/html/xen-devel/2008-11/msg00682.html Thanks, Yoshi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sat, Mar 07, 2009 at 02:26:06AM +0900, Yoshiaki Tamura wrote:> Although the numbers are old, you can get an extended abstract and slides > which I used at Xen Summit from http://www.osrg.net/kemariThanks, that has some more info. Seems like there''s a good opportunity for someone to do some more detailed comparisons.> >If both projects are indeed headed for the tree, people are going to > >need some ideas of which to use... > > Well, I sent the following message when I open-sourced Kemari in November. > > http://lists.xensource.com/archives/html/xen-devel/2008-11/msg00682.html"We''re also planning to merge with Remus from UBC to propose a better solution for Xen 3.4 together." That sounds very interesting. regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Friday, 06 March 2009 at 12:36, John Levon wrote:> On Sat, Mar 07, 2009 at 02:26:06AM +0900, Yoshiaki Tamura wrote: > > > Although the numbers are old, you can get an extended abstract and slides > > which I used at Xen Summit from http://www.osrg.net/kemari > > Thanks, that has some more info. Seems like there''s a good opportunity for > someone to do some more detailed comparisons. > > > >If both projects are indeed headed for the tree, people are going to > > >need some ideas of which to use... > > > > Well, I sent the following message when I open-sourced Kemari in November. > > > > http://lists.xensource.com/archives/html/xen-devel/2008-11/msg00682.html > > "We''re also planning to merge with Remus from UBC to propose a better > solution for Xen 3.4 together." > > That sounds very interesting.Yes, Yoshi and I had a brief discussion a few months ago about this. Kemari has been released, and I plan to release Remus as soon as possible. Unfortunately, I''ve had a very busy start to 2009 (related to the Tralfamadore project Andy presented at the last Xen summit) and haven''t had the cycles to forward port the code to unstable and test it. I intend to do that ASAP, and no later than the end of the month. The good news is that I believe all of the Xen and Linux support necessary for Remus has already been merged, so the code I''ll be releasing is just tools patches (I should double check whether that also holds for HVM Remus, but I believe it does). Once the code is released we can start looking at how to share code between Kemari and Remus. I think there is a lot of common ground between the two approaches -- the heart of Remus is fine-grained checkpoints, and I think that''s true for Kemari as well, though I haven''t yet looked closely at it. By the way, there''s another project going on here at UBC that complements both Kemari and Remus very well. Patrick Colp is working on copy-on-write VM checkpoints, which could be a huge win for checkpoint-based availability. He also presented some of his work at the last Xen summit: http://www.getxen.org/xensummit/xensummit_winter_2009.html _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Keir, On 2009/03/06 3:58, Keir Fraser wrote:> On 02/03/2009 13:30, "Yoshiaki Tamura" <tamura.yoshiaki@lab.ntt.co.jp> > wrote: > >> You can also get the undivided patches from the following links. >> http://www.osrg.net/kemari/download/kemari-v1-xen-20081120.patch >> http://www.osrg.net/kemari/download/kemari-v1-linux-20081120.patch > > The two obvious things I note are the duplicated save/restore functions, > which will simply bitrot, and the additions to evtchn.c. In particular theSince I have implemented the save/restore functions, will you let me ask some? I could not understand the meaning of "the duplicated save/restore functions". Do you mean there are some duplicated code inside xc_dom_kemari_save.c and xc_dom_kemari_restore.c? Or, do you mean there are some duplicated code, e.g. test_bit and clear_bit and so on, between xc_dom_kemari_save and xc_domain_save, and between xc_dom_kemari_restore and xc_domain_restore?> suspicious. Apart from that there are various spurious white-space and > debug-tracing changes which I assume you would clean up anyway. Those are > just the things that jump out from a quick read through.I will clean up those spurious white-space. Thank you in advance, -- Yoshisato YANAGISAWA, Ph.D. Open Source Software Computing Project NTT Cyber Space Laboratories NIPPON TELEGRAPH AND TELEPHONE CORPORATION 1-1 Hikarinooka, Yokosuka-Shi, Kanagawa 239-0847, Japan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 09/03/2009 02:05, "Yoshisato YANAGISAWA" <yanagisawa.yoshisato@lab.ntt.co.jp> wrote:> I could not understand the meaning of "the duplicated save/restore > functions". Do you mean there are some duplicated code inside > xc_dom_kemari_save.c and xc_dom_kemari_restore.c?xc_domain_{save,restore} and xc_dom_kemari_{save,restore} look to have quite a bit in common. It will be interesting to see Remus posted and compare patch sizes and approaches. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser wrote:> On 09/03/2009 02:05, "Yoshisato YANAGISAWA" > <yanagisawa.yoshisato@lab.ntt.co.jp> wrote: > >> I could not understand the meaning of "the duplicated save/restore >> functions". Do you mean there are some duplicated code inside >> xc_dom_kemari_save.c and xc_dom_kemari_restore.c? > > xc_domain_{save,restore} and xc_dom_kemari_{save,restore} look to have quite > a bit in common.Yes they do. We kept the original xc_domain_{save,restore} to make kemari specific part less intrusive. If you suggest to integrate them, we would certainly do so. Would that be appropriate? Yoshi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 09/03/2009 22:28, "Yoshiaki Tamura" <tamura.yoshiaki@lab.ntt.co.jp> wrote:>> xc_domain_{save,restore} and xc_dom_kemari_{save,restore} look to have quite >> a bit in common. > > Yes they do. We kept the original xc_domain_{save,restore} to make kemari > specific part less intrusive. If you suggest to integrate them, we would > certainly do so. Would that be appropriate?Yes, we prefer a merged save/restore codebase. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 2009/03/10 17:21, Keir Fraser wrote:> On 09/03/2009 22:28, "Yoshiaki Tamura" <tamura.yoshiaki@lab.ntt.co.jp> > wrote: > >>> xc_domain_{save,restore} and xc_dom_kemari_{save,restore} look to have quite >>> a bit in common. >> Yes they do. We kept the original xc_domain_{save,restore} to make kemari >> specific part less intrusive. If you suggest to integrate them, we would >> certainly do so. Would that be appropriate? > > Yes, we prefer a merged save/restore codebase.Since Kemari requires transaction-like feature, xc_dom_kemari_save should wait acknowledgment from xc_dom_kemari_restore. Xc_dom_kemari_restore should also buffer data until complete snapshot received from xc_dom_kemari_save. Because of this reason, there might be large if-else statements if those code are merged. Isn''t it make xc_domain_save and xc_domain_restore too complexed? -- Yoshisato YANAGISAWA, Ph.D. Open Source Software Computing Project NTT Cyber Space Laboratories NIPPON TELEGRAPH AND TELEPHONE CORPORATION 1-1 Hikarinooka, Yokosuka-Shi, Kanagawa 239-0847, Japan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 10/03/2009 10:16, "Yoshisato YANAGISAWA" <yanagisawa.yoshisato@lab.ntt.co.jp> wrote:>> Yes, we prefer a merged save/restore codebase. > > Since Kemari requires transaction-like feature, xc_dom_kemari_save > should wait acknowledgment from xc_dom_kemari_restore. > Xc_dom_kemari_restore should also buffer data until complete snapshot > received from xc_dom_kemari_save. > Because of this reason, there might be large if-else statements if those > code are merged. > > Isn''t it make xc_domain_save and xc_domain_restore too complexed?Well, you can leave it for now pending further investigation. I also want to see what the Remus approach is for this. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tuesday, 10 March 2009 at 10:38, Keir Fraser wrote:> On 10/03/2009 10:16, "Yoshisato YANAGISAWA" > <yanagisawa.yoshisato@lab.ntt.co.jp> wrote: > > >> Yes, we prefer a merged save/restore codebase. > > > > Since Kemari requires transaction-like feature, xc_dom_kemari_save > > should wait acknowledgment from xc_dom_kemari_restore. > > Xc_dom_kemari_restore should also buffer data until complete snapshot > > received from xc_dom_kemari_save. > > Because of this reason, there might be large if-else statements if those > > code are merged. > > > > Isn''t it make xc_domain_save and xc_domain_restore too complexed? > > Well, you can leave it for now pending further investigation. I also want to > see what the Remus approach is for this.The main change to xc_domain_save is to split out the code that does a round into its own function so that it can be called externally. I''ll have the patches together soon. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel