Xen users - Sep 2008 - Announcing the first release of SnowFlock

Hi everyone,
we''re releasing Snowflock to the general public. We''re making
a binary
and source relase, under the GNU General Public License (GPL). The 
release is available at http://compbio.cs.toronto.edu/snowflock.

Briefly, Snowflock lets you clone Xen VMs into dozens of identical 
replicas running in different hosts. Snowflock can do this in less than 
a second and with very low runtime overhead. With Snowflock you can, for 
example, perform parallel computations on the fly by scaling 
"instantaneously" your computing footprint in a shared cluster. 
Snowflock is a research prototype, hence the 0.1 major-minor. A minimum 
degree of experience with Xen and Linux is necessary to use the system. 
The contact address for snowflock is snowflock-users@cs.toronto.edu

More technically:
Snowflock is our prototype implementation of the / Impromptu Cluster 
(IC)/ abstraction. In an IC, an application encapsulated inside a 
virtual machine (VM) is swiftly forked into multiple copies that execute 
on different physical hosts, and then disappear when the computation 
ends. ICs simplify the development of parallel applications and reduces 
management burden by enabling the instantiation of new stateful 
computing elements: workers that need no setup time because they have a 
memory of the application state achieved up to the point of forking. 
This approach combines the benefits of cluster-based parallelism with 
those of running inside a VM.

Snowflock provides swift parallel VM cloning that makes it possible for 
Internet applications to deliver near-interactive performance for 
resource-intensive highly-parallelizable tasks. Snowflock makes use of 
four key techniques: /VM descriptors/ (condensed VM images that allow 
for sub-second suspension of a running VM and resumption of a of 
replicas); a /memory-on-demand/ subsystem that lazily populates the
VM''s
memory image during runtime; a set of / avoidance heuristics/ that 
minimize the amount of VM memory state to be fetched on demand; and a 
/multicast distribution/ system for commodity Ethernet networking 
hardware that makes the overhead of instantiating multiple VMs similar 
to that of instantiating a single one.

Use, enjoy, give us some feedback, and contribute to mankind :)
The Snowflock team


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, Sep 12, 2008 at 3:09 PM, Andres Lagar-Cavilla
<andreslc@cs.toronto.edu> wrote:
> More technically:
> Snowflock is our prototype implementation of the / Impromptu Cluster (IC)/
> abstraction. In an IC, an application encapsulated inside a virtual machine
> (VM) is swiftly forked into multiple copies that execute on different
> physical hosts, and then disappear when the computation ends. ICs simplify
> the development of parallel applications and reduces management burden by
> enabling the instantiation of new stateful computing elements: workers that
> need no setup time because they have a memory of the application state
> achieved up to the point of forking. This approach combines the benefits of
> cluster-based parallelism with those of running inside a VM.
>
> Snowflock provides swift parallel VM cloning that makes it possible for
> Internet applications to deliver near-interactive performance for
> resource-intensive highly-parallelizable tasks. Snowflock makes use of four
> key techniques: /VM descriptors/ (condensed VM images that allow for
> sub-second suspension of a running VM and resumption of a of replicas); a
> /memory-on-demand/ subsystem that lazily populates the VM''s memory
image
> during runtime; a set of / avoidance heuristics/ that minimize the amount
of
> VM memory state to be fetched on demand; and a /multicast distribution/
> system for commodity Ethernet networking hardware that makes the overhead
of
> instantiating multiple VMs similar to that of instantiating a single one.
just reading the docs.... i find it nice how you can drive the
duplication of VMs from within the VM.  great for grids.

could you elaborate about the memory-on-demand? i couldn''t find
anything about it on the manual, and it seems like a major advantage.

-- 
Javier

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

The more technical details of memory on demand etc can be found in our 
technical report: 
http://www.cs.toronto.edu/pub/reports/csrg/578/csrg-578-snowflock-LagarCavilla2008.pdf
Thanks for your interest!
Andres
Javier Guerra wrote:
> On Fri, Sep 12, 2008 at 3:09 PM, Andres Lagar-Cavilla
> <andreslc@cs.toronto.edu> wrote:
>   
>> More technically:
>> Snowflock is our prototype implementation of the / Impromptu Cluster
(IC)/
>> abstraction. In an IC, an application encapsulated inside a virtual
machine
>> (VM) is swiftly forked into multiple copies that execute on different
>> physical hosts, and then disappear when the computation ends. ICs
simplify
>> the development of parallel applications and reduces management burden
by
>> enabling the instantiation of new stateful computing elements: workers
that
>> need no setup time because they have a memory of the application state
>> achieved up to the point of forking. This approach combines the
benefits of
>> cluster-based parallelism with those of running inside a VM.
>>
>> Snowflock provides swift parallel VM cloning that makes it possible for
>> Internet applications to deliver near-interactive performance for
>> resource-intensive highly-parallelizable tasks. Snowflock makes use of
four
>> key techniques: /VM descriptors/ (condensed VM images that allow for
>> sub-second suspension of a running VM and resumption of a of replicas);
a
>> /memory-on-demand/ subsystem that lazily populates the VM''s
memory image
>> during runtime; a set of / avoidance heuristics/ that minimize the
amount of
>> VM memory state to be fetched on demand; and a /multicast distribution/
>> system for commodity Ethernet networking hardware that makes the
overhead of
>> instantiating multiple VMs similar to that of instantiating a single
one.
>>     
>
> just reading the docs.... i find it nice how you can drive the
> duplication of VMs from within the VM.  great for grids.
>
> could you elaborate about the memory-on-demand? i couldn''t find
> anything about it on the manual, and it seems like a major advantage.
>
>   

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel