Hello Everyone, I have an idea for setting up a virtualisation cluster, where the raw CPU power assigned to each DomU could be split across multiple physical servers. Does such a solution already exist? This would be fantastic for a "premimum" VPS hosting solutions where customer would be able to scale their VPSes to speeds and cores not possible with a single physical server. I appreciate that the latency that this would introduce would maybe make this project infeasible, nonetheless I''d be interested to hear everyone''s thoughts What would be even cooler if the RAM was distributed as well... Cheers _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jonathan Tripathy wrote:> > Hello Everyone, > > I have an idea for setting up a virtualisation cluster, where the raw > CPU power assigned to each DomU could be split across multiple > physical servers. > > Does such a solution already exist? This would be fantastic for a > "premimum" VPS hosting solutions where customer would be able to scale > their VPSes to speeds and cores not possible with a single physical > server. I appreciate that the latency that this would introduce would > maybe make this project infeasible, nonetheless I''d be interested to > hear everyone''s thoughts >You might want to google "single system image" and take a look at http://www.kerrighed.org Unfortunately, none of these seem to be compatible with Xen right now, though apparently some folks have been working along those lines. Xen on top of kerrighed would be way cool. Miles Fidelman -- In theory, there is no difference between theory and practice. In<fnord> practice, there is. .... Yogi Berra _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 06/04/2011 15:14, Miles Fidelman wrote:> Jonathan Tripathy wrote: >> >> Hello Everyone, >> >> I have an idea for setting up a virtualisation cluster, where the raw >> CPU power assigned to each DomU could be split across multiple >> physical servers. >> >> Does such a solution already exist? This would be fantastic for a >> "premimum" VPS hosting solutions where customer would be able to >> scale their VPSes to speeds and cores not possible with a single >> physical server. I appreciate that the latency that this would >> introduce would maybe make this project infeasible, nonetheless I''d >> be interested to hear everyone''s thoughts >> > > You might want to google "single system image" and take a look at > http://www.kerrighed.org > > Unfortunately, none of these seem to be compatible with Xen right now, > though apparently some folks have been working along those lines. Xen > on top of kerrighed would be way cool. > > Miles Fidelman > >Some userland VM monitor may work with that, such as virtualbox. What do Amazon do with their EC2 system? Surely those VMs aren''t running on a single physical box, are they? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Apr 6, 2011 at 9:20 AM, Jonathan Tripathy <jonnyt@abpni.co.uk> wrote:> Surely those VMs aren''t running on a single physical box, are they?sure they are. each physical box runs a small(ish) number of VMs, but no VM span more than a single box -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com [mailto:xen-users- > bounces@lists.xensource.com] On Behalf Of Miles Fidelman > Sent: Wednesday, April 06, 2011 10:15 AM > > You might want to google "single system image" and take a look at > http://www.kerrighed.orgDistributed shared memory is a hard problem (consider how you''d implement an efficient spinlock, for example). Google for "Cache-Coherent Distributed Shared Memory", e.g. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.83.8174&rep=rep 1&type=pdf. Even simple NUMA architectures are not well supported by modern software. Unfortunately it seems such machines are more theoretical than practical at this time. -Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 06/04/2011 16:00, Jeff Sturm wrote:>> -----Original Message----- >> From: xen-users-bounces@lists.xensource.com [mailto:xen-users- >> bounces@lists.xensource.com] On Behalf Of Miles Fidelman >> Sent: Wednesday, April 06, 2011 10:15 AM >> >> You might want to google "single system image" and take a look at >> http://www.kerrighed.org >> > Distributed shared memory is a hard problem (consider how you''d > implement an efficient spinlock, for example). Google for > "Cache-Coherent Distributed Shared Memory", e.g. > http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.83.8174&rep=rep > 1&type=pdf. > > Even simple NUMA architectures are not well supported by modern > software. Unfortunately it seems such machines are more theoretical > than practical at this time.Ah well. I''m still in deep thought about distributing raw CPU power across multiple machines for Xen DomUs. Does Amazon really only run their DomUs on single boxes? So their marketing that says "Elastic Computing" is just a simple stop DomU, change config file, resize lvm volume and restart DomU? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Apr 6, 2011, at 12:34 PM, Jonathan Tripathy wrote:> I''m still in deep thought about distributing raw CPU power across > multiple machines for Xen DomUs. Does Amazon really only run their > DomUs on single boxes? So their marketing that says "Elastic > Computing" is just a simple stop DomU, change config file, resize > lvm volume and restart DomU?You don''t really have to stop DomU to change all parameters -- a LVM resize would require it though. Anyway, EC2 does not even have persistent storage, so there''s no LVM there -- it''s all about the CPU and RAM. And increasing CPU and RAM does not require the DomU to be restarted (see "xm mem-set" and "xm vcpu-set"). The term "elastic" comes from the idea that you can virtually increase / decrease CPU and RAM according to your needs. It does not mean you can "stretch" a VM to a size larger than a single physical server and span it across multiple boxes. Their largest VM most likely takes an entire physical server. Best regards, Eduardo _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Virtualization does not work that way these days. Maybe it will eventually, though. It would definitely be another way to solve scalability issues. At this point in time, all of the major virtualization platforms limit each VM to one physical server. You cannot span a single VM across many physical servers. -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Jonathan Tripathy Sent: Wednesday, April 06, 2011 6:21 AM To: xen-users@lists.xensource.com Subject: [Xen-users] Clustering Hello Everyone, I have an idea for setting up a virtualisation cluster, where the raw CPU power assigned to each DomU could be split across multiple physical servers. Does such a solution already exist? This would be fantastic for a "premimum" VPS hosting solutions where customer would be able to scale their VPSes to speeds and cores not possible with a single physical server. I appreciate that the latency that this would introduce would maybe make this project infeasible, nonetheless I''d be interested to hear everyone''s thoughts What would be even cooler if the RAM was distributed as well... Cheers _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 06/04/2011 16:44, Eduardo Bragatto wrote:> > On Apr 6, 2011, at 12:34 PM, Jonathan Tripathy wrote: > >> I''m still in deep thought about distributing raw CPU power across >> multiple machines for Xen DomUs. Does Amazon really only run their >> DomUs on single boxes? So their marketing that says "Elastic >> Computing" is just a simple stop DomU, change config file, resize lvm >> volume and restart DomU? > > You don''t really have to stop DomU to change all parameters -- a LVM > resize would require it though. > > Anyway, EC2 does not even have persistent storage, so there''s no LVM > there -- it''s all about the CPU and RAM. And increasing CPU and RAM > does not require the DomU to be restarted (see "xm mem-set" and "xm > vcpu-set"). > > The term "elastic" comes from the idea that you can virtually increase > / decrease CPU and RAM according to your needs. It does not mean you > can "stretch" a VM to a size larger than a single physical server and > span it across multiple boxes. > >Well, it''s good to know that they aren''t doing any "magic" then :) (I''m guessing their persistent storage is the S3 side of things) BTW, they are offering you a VM with 64 cores: http://aws.amazon.com/hpc-applications/ Look on the left in the box that says "HPC Trial". Is this just a physical machines with lots of real CPUs? Thanks _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Apr 6, 2011 at 2:40 PM, Jonathan Tripathy <jonnyt@abpni.co.uk> wrote:> BTW, they are offering you a VM with 64 cores:no, they are not. that''s a video showing you how to "Build a 64-core cluster in under 10 minutes". that could be a cluster of 16 VMs with 4 cores each. it''s not a single VM -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 6 Apr 2011, at 21:30, Javier Guerra Giraldez <javier@guerrag.com> wrote:> On Wed, Apr 6, 2011 at 2:40 PM, Jonathan Tripathy <jonnyt@abpni.co.uk> wrote: >> BTW, they are offering you a VM with 64 cores: > > no, they are not. > > that''s a video showing you how to "Build a 64-core cluster in under 10 > minutes". that could be a cluster of 16 VMs with 4 cores each. it''s > not a single VM >Aah marketing! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hmm these sorts of architectures are possible. It''s worth looking at the work done by the TMEM guys on the xen-devel list. Technologies exist to mke the latency not such a big deal, for instance RDMA over Infiniband or SGI''s interconnect Numalink. We are still a few years off having this sort of capability but it is by no means impossible, just difficult and less efficient than running multiple machines at this time because of the really complex locking and sheduling that would need to be in place. Hope that helps. Joseph. On 7 April 2011 06:33, Jonathan Tripathy <jonnyt@abpni.co.uk> wrote:> > > > > On 6 Apr 2011, at 21:30, Javier Guerra Giraldez <javier@guerrag.com> wrote: > >> On Wed, Apr 6, 2011 at 2:40 PM, Jonathan Tripathy <jonnyt@abpni.co.uk> wrote: >>> BTW, they are offering you a VM with 64 cores: >> >> no, they are not. >> >> that''s a video showing you how to "Build a 64-core cluster in under 10 >> minutes". that could be a cluster of 16 VMs with 4 cores each. it''s >> not a single VM >> > Aah marketing! > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >-- Kind regards, Joseph. Founder | Director Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56 99 52 | Mobile: 0428 754 846 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Guys! Check out ScaleMP. This might be you''re looking for. They are also developing Xen/KVM over SSI. I don''t know if it is already released. Cheers, Jayson Cena _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users