Lustre discuss - Jul 2011 - Migrating virtual machines over Lustre using Proxmox

Like most of the readers on the list, my background with Lustre 
originates from cluster environments.  But as virtualization trends seem 
to be here to stay, the question of using Lustre to support large-scale 
distributed virtualization naturally arises.  Being able to leverage 
Lustre benefits in a VM cloud would seem to have quite a few advantages.

As a test case, at UNI we extended the Proxmox Virtualization 
Environment to support *live* Virtual Machine migration across separate 
physical (bare-metal) hosts of the Proxmox virtualization cluster, 
supported by a distributed Lustre filesystem.

If you aren''t familiar with Proxmox and live migration support over 
Lustre, what we deployed at UNI is akin to being able to do VMWare''s 
VMotion over Lustre (without the associated license costs).

We put together two screencasts showing the prototype deployment and 
wanted to share the proof-of-concept results with the community:

*)  A small demonstration of live migration with a small Debian VM whose 
root filesystem is supported over a distributed lustre implementation 
can be found here:
   http://dragon.cs.uni.edu/flash/proxmoxlustre.html

*)  A short screencast showing live migration over Lustre using the 
Proxmox GUI can be viewed here:
    http://dragon.cs.uni.edu/flash/gui-migration.html

Our immediate interests are in the performance of large (in terms of 
quantity), dynamic, live migrations that would leverage our 
high-throughput IB-based Lustre subsystem from our clusters.  We''d 
welcome your comments, feedback, questions or requests for specific 
benchmarks to explore.

ADVthanksANCE
-- 
Paul Gray                                         -o)
314 East Gym, Dept. of Computer Science           /\\
University of Northern Iowa                      _\_V
  Message void if penguin violated ...  Don''t mess with the penguin
  No one says, "Hey, I can''t read that ASCII attachment ya sent
me."

Hi Paul,

I wanted to signify our interest in your project as we have
something similar and related.

As part of the OSG ExTENCI project, we''ve set up a kerberized lustre
fs that uses virtual (VM) lustre clients in remote sites. With proper 
network tuning/route analysis, we observe that it is possible to
saturate the full IO bandwidth even for remote VM lustre clients
and obtain good IO rates.

So far, we''ve made available kerberized XEN VM lustre image 
(ftp://ftp.psc.edu/pub/jwan/Lustre-2.1/2.0.62/vm-images/) that
ExTENCI Tier3 remote sites can download and just boot up after
being given the proper kerberos principals.

We will also provide the kerberized images for KVM (Proxmox)
and VMware.

Currently, we use Lustre 2.1 (2.0.62) with 2.0.63 for clients.
PSC locally runs the same set up on a separate kerberos realm.

We invite collaboration with other parties who might be
interested in trying packaged kerberized lustre VM  clients
at their  sites.

Regards,
josephine



On Sat, 9 Jul 2011, Paul Gray wrote:
> Like most of the readers on the list, my background with Lustre
> originates from cluster environments.  But as virtualization trends seem
> to be here to stay, the question of using Lustre to support large-scale
> distributed virtualization naturally arises.  Being able to leverage
> Lustre benefits in a VM cloud would seem to have quite a few advantages.
>
> As a test case, at UNI we extended the Proxmox Virtualization
> Environment to support *live* Virtual Machine migration across separate
> physical (bare-metal) hosts of the Proxmox virtualization cluster,
> supported by a distributed Lustre filesystem.
>
> If you aren''t familiar with Proxmox and live migration support
over
> Lustre, what we deployed at UNI is akin to being able to do
VMWare''s
> VMotion over Lustre (without the associated license costs).
>
> We put together two screencasts showing the prototype deployment and
> wanted to share the proof-of-concept results with the community:
>
> *)  A small demonstration of live migration with a small Debian VM whose
> root filesystem is supported over a distributed lustre implementation
> can be found here:
>   http://dragon.cs.uni.edu/flash/proxmoxlustre.html
>
> *)  A short screencast showing live migration over Lustre using the
> Proxmox GUI can be viewed here:
>    http://dragon.cs.uni.edu/flash/gui-migration.html
>
> Our immediate interests are in the performance of large (in terms of
> quantity), dynamic, live migrations that would leverage our
> high-throughput IB-based Lustre subsystem from our clusters.  We''d
> welcome your comments, feedback, questions or requests for specific
> benchmarks to explore.
>
> ADVthanksANCE
> -- 
> Paul Gray                                         -o)
> 314 East Gym, Dept. of Computer Science           /\\
> University of Northern Iowa                      _\_V
>  Message void if penguin violated ...  Don''t mess with the penguin
>  No one says, "Hey, I can''t read that ASCII attachment ya
sent me."
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>