Hi all, Has anyone created a Xen cluster to run a hadoop vm cluster? I would be interested in how it performs Thanks Lance -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
We are working on MogileFS cloud storage over Xen! We''re going to store documents of small size ~1 MB. What is your business case? Regards Rajan On Fri, Nov 6, 2009 at 5:01 PM, Lance Haig <lhaig@haigmail.com> wrote:> Hi all, > > > Has anyone created a Xen cluster to run a hadoop vm cluster? > > I would be interested in how it performs > > Thanks > > Lance > > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi rajan, We currently run a 10 hadoop cluster on 3 vmware hosts and I want to create a bigger cluster on a xen cluster of servers. so I was just wanting to find out how easy it is. Our HD cluster maxes out the vmware CPUs for about 4 Hours a day doing our analysis. So I think your answer says it all for me. It is possible so we should test. Thanks Lance rajan chandi wrote:> We are working on MogileFS cloud storage over Xen! > > We''re going to store documents of small size ~1 MB. > > What is your business case? > > Regards > Rajan > > On Fri, Nov 6, 2009 at 5:01 PM, Lance Haig <lhaig@haigmail.com > <mailto:lhaig@haigmail.com>> wrote: > > Hi all, > > > Has anyone created a Xen cluster to run a hadoop vm cluster? > > I would be interested in how it performs > > Thanks > > Lance > > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com <mailto:Xen-users@lists.xensource.com> > http://lists.xensource.com/xen-users > > > > -- > This message has been scanned for viruses and > dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is > believed to be clean. > > > ------------------------------------------------------------------------ > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users-- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Lance, It''s not hard to create the cluster. We are using Xen over Debian Lenny. We''re yet to release our SAAS product in production. So, We''ve not load tested it yet. What were your decision factors in choosing HD over other File Systems? Our research states that MogileFS is more scalable and good for small files..while HDFS is good for large files. Regards Rajan On Mon, Nov 9, 2009 at 1:35 AM, Lance Haig <lhaig@haigmail.com> wrote:> Hi rajan, > > We currently run a 10 hadoop cluster on 3 vmware hosts and I want to create > a bigger cluster on a xen cluster of servers. so I was just wanting to find > out how easy it is. Our HD cluster maxes out the vmware CPUs for about 4 > Hours a day doing our analysis. > > So I think your answer says it all for me. It is possible so we should > test. > > Thanks > > Lance > > rajan chandi wrote: > >> We are working on MogileFS cloud storage over Xen! >> >> We''re going to store documents of small size ~1 MB. >> >> What is your business case? >> >> Regards >> Rajan >> >> On Fri, Nov 6, 2009 at 5:01 PM, Lance Haig <lhaig@haigmail.com <mailto: >> lhaig@haigmail.com>> wrote: >> >> Hi all, >> >> >> Has anyone created a Xen cluster to run a hadoop vm cluster? >> >> I would be interested in how it performs >> >> Thanks >> >> Lance >> >> >> -- >> This message has been scanned for viruses and >> dangerous content by MailScanner, and is >> believed to be clean. >> >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com <mailto:Xen-users@lists.xensource.com> >> >> http://lists.xensource.com/xen-users >> >> >> >> -- >> This message has been scanned for viruses and >> dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is >> >> believed to be clean. >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> > > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
The part that can be problematic is the ones that involve I/O - network and disk. You have to configure network interface in a way that each domU VM with Hadoop instances on a single dom0 don''t fight for a single physical network interface, as it is possible that lots of data could be moved around in between Hadoop Datanodes. You probably have to install more than one network interface so that each VM can have a dedicated one. Same goes for the disk I/O - you don''t want domU''s on a single dom0 fight for accesses to a single disk. Make sure you distribute disk accesses among domU''s... These issues aside, it''s a clever way to deploy Hadoop. Amazon offers EC2 with Hadoop, and I think it''s pretty much the same as what you are trying to do. /Taeho On Mon, Nov 9, 2009 at 7:09 PM, rajan chandi <chandi.rajan@gmail.com> wrote:> > Hi Lance, > > It''s not hard to create the cluster. We are using Xen over Debian Lenny. > > We''re yet to release our SAAS product in production. > > So, We''ve not load tested it yet. > > What were your decision factors in choosing HD over other File Systems? > > Our research states that MogileFS is more scalable and good for small > files..while HDFS is good for large files. > > Regards > Rajan > > > On Mon, Nov 9, 2009 at 1:35 AM, Lance Haig <lhaig@haigmail.com> wrote: > >> Hi rajan, >> >> We currently run a 10 hadoop cluster on 3 vmware hosts and I want to >> create a bigger cluster on a xen cluster of servers. so I was just wanting >> to find out how easy it is. Our HD cluster maxes out the vmware CPUs for >> about 4 Hours a day doing our analysis. >> >> So I think your answer says it all for me. It is possible so we should >> test. >> >> Thanks >> >> Lance >> >> rajan chandi wrote: >> >>> We are working on MogileFS cloud storage over Xen! >>> >>> We''re going to store documents of small size ~1 MB. >>> >>> What is your business case? >>> >>> Regards >>> Rajan >>> >>> On Fri, Nov 6, 2009 at 5:01 PM, Lance Haig <lhaig@haigmail.com <mailto: >>> lhaig@haigmail.com>> wrote: >>> >>> Hi all, >>> >>> >>> Has anyone created a Xen cluster to run a hadoop vm cluster? >>> >>> I would be interested in how it performs >>> >>> Thanks >>> >>> Lance >>> >>> >>> -- >>> This message has been scanned for viruses and >>> dangerous content by MailScanner, and is >>> believed to be clean. >>> >>> >>> _______________________________________________ >>> Xen-users mailing list >>> Xen-users@lists.xensource.com <mailto:Xen-users@lists.xensource.com> >>> >>> http://lists.xensource.com/xen-users >>> >>> >>> >>> -- >>> This message has been scanned for viruses and >>> dangerous content by *MailScanner* <http://www.mailscanner.info/>, and >>> is >>> >>> believed to be clean. >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> Xen-users mailing list >>> Xen-users@lists.xensource.com >>> http://lists.xensource.com/xen-users >>> >> >> >> -- >> This message has been scanned for viruses and >> dangerous content by MailScanner, and is >> believed to be clean. >> >> > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users