Two quick Questions. Does lustre work in a VM? what about in a VM over Infiniband? Brian O''Connor ------------------------------------------------------------------------------- SGI Consulting Email: briano at sgi.com, Mobile +61 417 746 452 Phone: +61 3 9963 1900, Fax: +61 3 9963 1902 357 Camberwell Road, Camberwell, Victoria, 3124 AUSTRALIA http://www.sgi.com/support/services ------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100908/c34daac4/attachment.html
On Wed, 2010-09-08 at 05:50 -0500, Brian O''Connor wrote:> Does lustre work in a VM?Yes, of course, given that a VM provides an entire virtual computer.> what about in a VM over Infiniband?I don''t know of any VMs which expose the hosts Infiniband hardware for the VM to use directly. Xen might. libvirt/kvm might. But those are just WAGs. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100908/c99a7816/attachment.bin
For training porpuse we use virtualbox. We have a patchless client in production on a xen virtual machine with 10GbE without problem. Bye On 09/08/2010 02:09 PM, Brian J. Murrell wrote:> On Wed, 2010-09-08 at 05:50 -0500, Brian O''Connor wrote: > >> Does lustre work in a VM? >> > Yes, of course, given that a VM provides an entire virtual computer. > > >> what about in a VM over Infiniband? >> > I don''t know of any VMs which expose the hosts Infiniband hardware for > the VM to use directly. Xen might. libvirt/kvm might. But those are > just WAGs. > > b. > > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >-- _Gabriele Paciucci_ http://www.linkedin.com/in/paciucci Pursuant to legislative Decree n. 196/03 you are hereby informed that this email contains confidential information intended only for use of addressee. If you are not the addressee and have received this email by mistake, please send this email to the sender. You may not copy or disseminate this message to anyone. Thank You.
On 9/8/10 8:09 AM, Brian J. Murrell wrote:> On Wed, 2010-09-08 at 05:50 -0500, Brian O''Connor wrote: >> Does lustre work in a VM? > > Yes, of course, given that a VM provides an entire virtual computer. > >> what about in a VM over Infiniband? > > I don''t know of any VMs which expose the hosts Infiniband hardware for > the VM to use directly. Xen might. libvirt/kvm might. But those are > just WAGs.AFAIK, Xen can only expose IB HCAs to guests via PCI passthrough: http://wiki.xensource.com/xenwiki/XenPCIpassthrough Mellanox has a modified OFED 1.3.1 for VMWare that provides "Virtual-IQ", never tried it myself though: http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=36&menu_section=34 You can bridge IPoIB interfaces into guests just like any other network device (and thus socklnd). You can export SRP block devices to guests just like any other block device (e.g. for OST/MDT). Ideally, someone would implement SR-IOV support in OFED: http://www.pcisig.com/specifications/iov/ Which works nicely for, e.g. igb. Lots of talk about IB virtualization a few years ago, unfortunately doesn''t appear to be much recent work in this area that I can find. - Dardo
I seem to recall Mellanox presenting a paper on IB support virtual machines at SC two years ago. I think it was just a proof of concept, and I''m unaware of the current status. Kevin On Sep 8, 2010, at 6:09 AM, "Brian J. Murrell" <brian.murrell at oracle.com> wrote:> On Wed, 2010-09-08 at 05:50 -0500, Brian O''Connor wrote: >> Does lustre work in a VM? > > Yes, of course, given that a VM provides an entire virtual computer. > >> what about in a VM over Infiniband? > > I don''t know of any VMs which expose the hosts Infiniband hardware for > the VM to use directly. Xen might. libvirt/kvm might. But those are > just WAGs. > > b. > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
Am 08.09.10 14:09, schrieb Brian J. Murrell: > On Wed, 2010-09-08 at 05:50 -0500, Brian O''Connor wrote: >> what about in a VM over Infiniband? > I don''t know of any VMs which expose the hosts Infiniband hardware for > the VM to use directly. Xen might. libvirt/kvm might. But those are > just WAGs. > > b. At the last International Supercomputing Conference Mellanox claimed to have an Infiniband solution ready to support Infiniband in Virtual Machines with nearly no performance impact. It is based on the SR-IOV Technologie. Some Information can be found in their blog at: http://www.mellanox.com/blog/2009/04/io-virtualization/ If someone already uses this technology it would be nice to get some feedback how well it is working. Cheers, Florian Feldhaus -- ------------------------------------------------------------------------------- Florian Feldhaus | voice: +49-231-755 5324 ITMC/LS Service Computing | fax: +49-231-755 2731 TU Dortmund | office: GB 5, 347 D-44227 Dortmund | email: florian.feldhaus at tu-dortmund.de Germany | http://www.itmc.tu-dortmund.de/ -------------------------------------------------------------------------------
Mellanox sold us "ConnectX-2" IB cards earlier this year which claim to not only work on a single VM, passed through via pci pass-through, but also to be able to be shared between multiple VM''s. Vmware support was supposed to be there at the time, then KVM and Xen. We are already running Lustre under KVM on our cloud cluster but we have not yet had time to try the infiniband feature to see if we can see the infiniband cards in the VM. But the vanilla Xen kernel on the bare metal does recognize the infiniband card so that is a good sign. (PS--the Lustre-under-KVM is a proof-of-concept only at this stage, obviously you do not get the kind of throughput that you get on bare metal.) Steve On Wed, 8 Sep 2010, Kevin Van Maren wrote:> I seem to recall Mellanox presenting a paper on IB support virtual > machines at SC two years ago. I think it was just a proof of concept, > and I''m unaware of the current status. > > Kevin > > > On Sep 8, 2010, at 6:09 AM, "Brian J. Murrell" > <brian.murrell at oracle.com> wrote: > >> On Wed, 2010-09-08 at 05:50 -0500, Brian O''Connor wrote: >>> Does lustre work in a VM? >> >> Yes, of course, given that a VM provides an entire virtual computer. >> >>> what about in a VM over Infiniband? >> >> I don''t know of any VMs which expose the hosts Infiniband hardware for >> the VM to use directly. Xen might. libvirt/kvm might. But those are >> just WAGs. >> >> b. >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >-- ------------------------------------------------------------------ Steven C. Timm, Ph.D (630) 840-8525 timm at fnal.gov http://home.fnal.gov/~timm/ Fermilab Computing Division, Scientific Computing Facilities, Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader.