We recently developed XenLoop -- a transparent inter-VM network loopback mechanism. XenLoop allows unmodified network applications to bypass the standard network data path via Dom0 when talking with other guest VMs on the same machine. We''d very much like to hear usage feedback and suggestions from others interested in this work. XenLoop code and documentation is available here: http://osnet.cs.binghamton.edu/projects/xenloop.html XenLoop differs from earlier approaches (such as XenSockets, IVC, and XWay) as follows: - No changes, such as recompiling or relinking, are needed to the user applications or libraries. - XenLoop doesn''t modify any part of the core kernel code - XenLoop transparently adapts to migration without disrupting any ongoing network communication and without user or application intervention. - Being implemented between the network and data-link layers (using netfilter hook), XenLoop can currently handle all IPv4 traffic, and likely others in near future. - On the flip side, XenLoop presently incurs the TCP/IP protocol processing cost in each guest, which other approaches can avoid to obtain higher throughput while trading off some application-level or kernel-level transparency. We still observe quite a decent improvement in inter-VM communication performance on dual-core machines when compared to the Dom0 data path -- up to 5X reduction in RTT and up to 6X increase in throughput. We''ve tested XenLoop on Xen 3.1 and Linux 2.6.18. Next steps are to try it on Xen 3.2 and to possibly move traffic interception higher up the protocol stack. Please contact Jian Wang (jianwang@cs.binghamton.edu) and Kartik Gopalan (kartik@cs.binghamton.edu). -- Kartik Gopalan Assistant Professor, Computer Science Thomas J. Watson School of Engineering and Applied Sciences State University of New York at Binghamton kartik@cs.binghamton.edu http://www.cs.binghamton.edu/~kartik _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sat, 2008-02-09 at 21:07 -0500, Kartik Gopalan wrote:> We recently developed XenLoop -- a transparent inter-VM > network loopback mechanism. XenLoop allows unmodified > network applications to bypass the standard > network data path via Dom0 when talking with > other guest VMs on the same machine.I''m not yet through with it, and promise I''ll try to deliver more useful comments soon. But here''s the first thing which kinda bewilders me: A loopback sends packets back to the originator. Hence the name. You propose point-to-point links. A bunch of virtual crossover cables, similar to the original association between backends and frontends. So naming it ''XenLoop'' is kinda misleading. kind regards, daniel PS: I hereby admit that XenCross sounds plain stupid. :) -- Daniel Stodden LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation Institut für Informatik der TU München D-85748 Garching http://www.lrr.in.tum.de/~stodden mailto:stodden@cs.tum.edu PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
The analogy, though not perfect, is that a loopback carries traffic between different network endpoints within the same (virtual) machine -- the endpoints may be in different processes, not just the originator. XenLoop carries traffic between different network endpoints resident in the same "physical" machine but in different VMs. Your virtual crossover cable analogy is just how it happens to be currently implemented. One could implement XenLoop using other non-crossover-type designs, resulting in more VM crosstalk or less security, but functionally what it does would remain unchanged, which is to bypass the network datapath through Dom0. - Kartik On Feb 10, 2008 2:41 PM, Daniel Stodden <stodden@cs.tum.edu> wrote:> > On Sat, 2008-02-09 at 21:07 -0500, Kartik Gopalan wrote: > > We recently developed XenLoop -- a transparent inter-VM > > network loopback mechanism. XenLoop allows unmodified > > network applications to bypass the standard > > network data path via Dom0 when talking with > > other guest VMs on the same machine. > > I''m not yet through with it, and promise I''ll try to deliver more useful > comments soon. But here''s the first thing which kinda bewilders me: > > A loopback sends packets back to the originator. Hence the name. You > propose point-to-point links. A bunch of virtual crossover cables, > similar to the original association between backends and frontends. So > naming it ''XenLoop'' is kinda misleading. > > kind regards, > daniel > > PS: I hereby admit that XenCross sounds plain stupid. :) > > -- > Daniel Stodden > LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation > Institut für Informatik der TU München D-85748 Garching > http://www.lrr.in.tum.de/~stodden mailto:stodden@cs.tum.edu > PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, 2008-02-11 at 21:51 -0500, Kartik Gopalan wrote:> The analogy, though not perfect, is that a loopback carries traffic > between different network endpoints within the same (virtual) > machine -- the endpoints may be in different processes, not just > the originator. XenLoop carries traffic between different network > endpoints resident in the same "physical" machine but in > different VMs. Your virtual crossover cable analogy is just how > it happens to be currently implemented. One could implement > XenLoop using other non-crossover-type designs, resulting in > more VM crosstalk or less security, but functionally what it does > would remain unchanged, which is to bypass the network > datapath through Dom0.I retreat. :) Just found that it does not reflect the difference to the dom0 switch, which functionally qualifies as a loopback by the same definition. And it''s certainly not meant to question your work. I''ve got a couple of additional questions and comments. 1. Netfilter setup: You''re hooking into the output chain, just before the packet is passed to the guest eth0? Did I get this right, or something more esoteric? I''m asking because I''m definitely no netfilter guru. 2. Periodic scanning of XenStore entries: Why not callbacks? 3. Announcement messages: This is a layer 3 (IP) packet? I''d rather expected an ethertype. 4. Personally (this is dicussible) I don''t like the XenStore/Message *mixture* the directory is maintained with. I agree that due to the fact that guests are limited to their private subtrees, maintenance of a common directory is non-obvious but this is a defect in XenStore if the directory should be there. XenStore is obvious, messages dissemination as well. A dedicated resolution protocol (i.e. a domain-arp) appears cleaner to me. I suppose there are issues in getting this to work cleanly during migration (i.e. the need for invalidation messages before leaving the host), or why did you choose this design? Still not through, therefore likely more to come. Thanks, Daniel -- Daniel Stodden LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation Institut für Informatik der TU München D-85748 Garching http://www.lrr.in.tum.de/~stodden mailto:stodden@cs.tum.edu PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> 1. Netfilter setup: You''re hooking into the output chain, just before > the packet is passed to the guest eth0? Did I get this right, or > something more esoteric? I''m asking because I''m definitely no netfilter > guru.Yes, basically call nf_register_hook on NF_IP_POST_ROUTING> 2. Periodic scanning of XenStore entries: Why not callbacks?One could...we just happened to implement it as a simple thread to poll all active domains periodically in one thread and send out announcements. This was not in the critical path.> 3. Announcement messages: This is a layer 3 (IP) packet? I''d rather > expected an ethertype.No, its a layer 3 XenLoop-type packet, not IP -- the new protocol type for exchanging control messages is registered at the same level as IP.> 4. Personally (this is dicussible) I don''t like the XenStore/Message > *mixture* the directory is maintained with. I agree that due to the fact > that guests are limited to their private subtrees, maintenance of a > common directory is non-obvious but this is a defect in XenStore if the > directory should be there. XenStore is obvious, messages dissemination > as well. A dedicated resolution protocol (i.e. a domain-arp) appears > cleaner to me. I suppose there are issues in getting this to work > cleanly during migration (i.e. the need for invalidation messages before > leaving the host), or why did you choose this design?I am not sure, I understood the question clearly, but I''ll try. XenStore is used here mainly for 3 things: - For a guest to advertise its intent to participate in XenLoop to Domain 0 - For Domain 0 to read the guest''s intent and announce it to other interested guests - For catching pre/post migration notices This seems to work reasonably efficiently for now. One reason to have Domain0 participate in guest discovery is the limitation that one guest cannot read another''s XenStore entries directly (with good reason) to discover co-resident guests. There also doesn''t seem to be any notion yet of a machine-local broadcast message (which may have other drawbacks) or even a machine-local multicast group. As for domain-arp, there''s the question of do we really want any random guest to be able to discover any other guest on the same machine? Using XenStore, with Domain 0 as arbiter, provides some level of control over who gets to see whom. As for why we designed it this way, because it happens to work cleanly...we tried initially to just broadcast messages from each guest, but didn''t know how to contain the broadcasts to be machine-local.> Still not through, therefore likely more to come.Certainly. Thanks for the comments. - Kartik _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel