Hi, Today we have a storage system based on NFS, but we are really concerned about redundancy and are at the brink to take the step to a cluster file system as glusterfs, but we have got suggestions on that lusterfs would have been the best option for us, but at the same time those who "recommended" lusterfs has said that Oracle has pulled the plug and put the resources into OCFS2. If using lusterfs in a production environment, it would be good to know that it won''t be discontinued. Will there be a long term future for lusterfs? Or should we be looking for something else for a long term solution? Thanks in advance for your reply for my a bit cloudy question. -- Janne Aho (Developer) | City Network Hosting AB - www.citynetwork.se Phone: +46 455 690022 | Cell: +46 733 312775 EMail/MSN: janne at citynetwork.se ICQ: 567311547 | Skype: janne_mz | AIM: janne4cn | Gadu: 16275665
On 2010-04-22, at 00:33, Janne Aho wrote:> Today we have a storage system based on NFS, but we are really concerned > about redundancy and are at the brink to take the step to a cluster file > system as glusterfs, but we have got suggestions on that lusterfs would > have been the best option for us, but at the same time those who > "recommended" lusterfs has said that Oracle has pulled the plug and put > the resources into OCFS2.Whoever told you that is mis-informed. We still have our full team developing Lustre at Oracle, and are planning development into the future.> If using lusterfs in a production environment, it would be good to know > that it won''t be discontinued. > > Will there be a long term future for lusterfs?Yes. Cheers, Andreas -- Andreas Dilger Principal Engineer, Lustre Group Oracle Corporation Canada Inc.
Am Donnerstag, 22. April 2010 08:33:14 schrieb Janne Aho:> Hi, > > Today we have a storage system based on NFS, but we are really concerned > about redundancy and are at the brink to take the step to a cluster file > system as glusterfs, but we have got suggestions on that lusterfs would > have been the best option for us, but at the same time those who > "recommended" lusterfs has said that Oracle has pulled the plug and put > the resources into OCFS2. > If using lusterfs in a production environment, it would be good to know > that it won''t be discontinued. > > Will there be a long term future for lusterfs? > Or should we be looking for something else for a long term solution? > > Thanks in advance for your reply for my a bit cloudy question.Hi, for me Lustre is a very good option. But you also could consider a system composed from - corosync for the cluster communication - pacemaker as a cluster resource manager - DRBD for the replication of data between nodes in a cluster and - NFS or - OCFS2 or GFS or ... especially the NFS option provides you with a high available NFS server on real cluster stack all managed by pacemaker. Greetings, -- Dr. Michael Schwartzkopff MultiNET Services GmbH Addresse: Bretonischer Ring 7; 85630 Grasbrunn; Germany Tel: +49 - 89 - 45 69 11 0 Fax: +49 - 89 - 45 69 11 21 mob: +49 - 174 - 343 28 75 mail: misch at multinet.de web: www.multinet.de Sitz der Gesellschaft: 85630 Grasbrunn Registergericht: Amtsgericht M?nchen HRB 114375 Gesch?ftsf?hrer: G?nter Jurgeneit, Hubert Martens --- PGP Fingerprint: F919 3919 FF12 ED5A 2801 DEA6 AA77 57A4 EDD8 979B Skype: misch42
On 22/04/10 08:56, Michael Schwartzkopff wrote: On 22/04/10 08:53, Andreas Dilger wrote:> Am Donnerstag, 22. April 2010 08:33:14 schrieb Janne Aho: >> Hi, >> >> Today we have a storage system based on NFS, but we are really concerned >> about redundancy and are at the brink to take the step to a cluster file >> system as glusterfs, but we have got suggestions on that lusterfs would >> have been the best option for us, but at the same time those who >> "recommended" lusterfs has said that Oracle has pulled the plug and put >> the resources into OCFS2. >> If using lusterfs in a production environment, it would be good to know >> that it won''t be discontinued.On 22/04/10 08:53, Andreas Dilger wrote:> Whoever told you that is mis-informed. We still have our full team developing Lustre at Oracle, and are planning development into the future.Andreas, thanks for the reply. :)>> Will there be a long term future for lusterfs? >> Or should we be looking for something else for a long term solution? >> >> Thanks in advance for your reply for my a bit cloudy question.On 22/04/10 08:56, Michael Schwartzkopff wrote:> for me Lustre is a very good option. > > But you also could consider a system composed from > - corosync for the cluster communication > - pacemaker as a cluster resource manager > - DRBD for the replication of data between nodes in a clusterMichael, thanks for the advice, we will take a big dive into this.> and > - NFS > or > - OCFS2 or GFS or ... > > especially the NFS option provides you with a high available NFS server on > real cluster stack all managed by pacemaker.We have had quite a deal of issues with our setup using NFS, one of the problems has been caused by poor file system selection thanks to the poor selection of file systems in CentOS/RedHat EL. -- Janne Aho (Developer) | City Network Hosting AB - www.citynetwork.se Phone: +46 455 690022 | Cell: +46 733 312775 EMail/MSN: janne at citynetwork.se ICQ: 567311547 | Skype: janne_mz | AIM: janne4cn | Gadu: 16275665
Make sure you read and understand the Lustre 2.0 release notes before you buy. There seemed to be some specifics in there about using Oracle hardware. -----Original Message----- From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Andreas Dilger Sent: Thursday, April 22, 2010 12:53 AM To: Janne Aho Cc: lusterfs Subject: Re: [Lustre-discuss] Future of LusterFS? On 2010-04-22, at 00:33, Janne Aho wrote:> Today we have a storage system based on NFS, but we are really concerned > about redundancy and are at the brink to take the step to a cluster file > system as glusterfs, but we have got suggestions on that lusterfs would > have been the best option for us, but at the same time those who > "recommended" lusterfs has said that Oracle has pulled the plug and put > the resources into OCFS2.Whoever told you that is mis-informed. We still have our full team developing Lustre at Oracle, and are planning development into the future.> If using lusterfs in a production environment, it would be good to know > that it won''t be discontinued. > > Will there be a long term future for lusterfs?Yes. Cheers, Andreas -- Andreas Dilger Principal Engineer, Lustre Group Oracle Corporation Canada Inc. _______________________________________________ Lustre-discuss mailing list Lustre-discuss at lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
>Make sure you read and understand the Lustre 2.0 release notes before you >buy. There seemed to be some specifics in there about using Oracle hardware.In all fairness ... that only matters if you pay Oracle for support. If you aren''t paying Oracle for support (or have no plans to), then it doesn''t matter. --Ken
If you read the presentation, at http://wiki.lustre.org/images/6/6f/LUG_Keynote_Presentation-Bojanic-100415.pdf it does say that Lustre 2 will be supported on qualified configurations -- from Oracle and others. Kevin Lundgren, Andrew wrote:> Make sure you read and understand the Lustre 2.0 release notes before you buy. There seemed to be some specifics in there about using Oracle hardware. > > -----Original Message----- > From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Andreas Dilger > Sent: Thursday, April 22, 2010 12:53 AM > To: Janne Aho > Cc: lusterfs > Subject: Re: [Lustre-discuss] Future of LusterFS? > > On 2010-04-22, at 00:33, Janne Aho wrote: > >> Today we have a storage system based on NFS, but we are really concerned >> about redundancy and are at the brink to take the step to a cluster file >> system as glusterfs, but we have got suggestions on that lusterfs would >> have been the best option for us, but at the same time those who >> "recommended" lusterfs has said that Oracle has pulled the plug and put >> the resources into OCFS2. >> > > Whoever told you that is mis-informed. We still have our full team developing Lustre at Oracle, and are planning development into the future. > > >> If using lusterfs in a production environment, it would be good to know >> that it won''t be discontinued. >> >> Will there be a long term future for lusterfs? >> > > Yes. > > Cheers, Andreas > -- > Andreas Dilger > Principal Engineer, Lustre Group > Oracle Corporation Canada Inc. > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
On 22/04/10 17:38, Lundgren, Andrew wrote: (somehow managed to send this as private mail, while it was ment to be sent to the list) sorry being old fashioned and answer inline, but it feels less jeopardy.> I think the lustre 2.0 release notes indicated that lustre will continue but may only be supported on Oracle hardware by Oracle. > If you are doing anything else, it seemed like you would be on your own.In our economical calculation there ain''t much space for support fees and we would getting some 3rd party support from one of our partners.> That said, http://www.clusterstor.com/ is a new company founded by Peter Braum (the guy who invented Lustre). > They are creating a new cluster file system as well as supporting Lustre. They have a customers link off of their website that indicates some of the notables.Interesting, but a bit hefty price tag for us.> There is a possibility that there will be a lustre fork in the future. > Some following Oracle''s "opensource" model and the other following the more traditional model.After reading "After the Software wars" by Keith Curtis, in the long run, I think I''ll be betting on the open source project than the closed one. We are still here talking a bit about LustreFS vs GlusterFS, as the first time we will be using a cluster file system it feels quite difficult what to choose and the same time we need to keep the total cost as low as possible. Would lustre have issues if using cheap off the shell components or would people here think you need to have high end maskines with built in redundancy for everything? -- Janne Aho (Developer) | City Network Hosting AB - www.citynetwork.se Phone: +46 455 690022 | Cell: +46 733 312775 EMail/MSN: janne at citynetwork.se ICQ: 567311547 | Skype: janne_mz | AIM: janne4cn | Gadu: 16275665
We run lustre on cheap off the shelf gear. We have 4 generations of cheapish gear in a single 300TB lustre config (40 oss''s) It has been running very very well for about 3.5 years now.> Would lustre have issues if using cheap off the shell components or > would people here think you need to have high end maskines with built in > redundancy for everything?-- Dr Stuart Midgley sdm900 at gmail.com
On 23/04/10 11:42, Stu Midgley wrote:>> Would lustre have issues if using cheap off the shell components or >> would people here think you need to have high end machines with built in >> redundancy for everything? > > We run lustre on cheap off the shelf gear. We have 4 generations of > cheapish gear in a single 300TB lustre config (40 oss''s) > > It has been running very very well for about 3.5 years now.This sounds promising. Have you had any hardware failures? If yes, how well has the cluster cooped with the loss of the machine(s)? Any advice you can share from your initial setup of lustre? -- Janne Aho (Developer) | City Network Hosting AB - www.citynetwork.se Phone: +46 455 690022 | Cell: +46 733 312775 EMail/MSN: janne at citynetwork.se ICQ: 567311547 | Skype: janne_mz | AIM: janne4cn | Gadu: 16275665
Taking a break from my current non-computer related work.. My guess based on your success is your gear is not so much cheap, as *cost effective high MTBF commodity parts*. If you go for the absolute bargain basement stuff, you''ll have problems as individual components flake out. If you spend way too much money on high-end multi-redundant whizbangs, you generally get two things.. redundancy, which in my mind often only serves to make the eventual failure worse, and high-quality, long MTBF components. If you can get the high MTBF components without all the redudancy (and associated complexity nightmare), then you win. On Fri, Apr 23, 2010 at 05:42:30PM +0800, Stu Midgley wrote:> We run lustre on cheap off the shelf gear. We have 4 generations of > cheapish gear in a single 300TB lustre config (40 oss''s) > > It has been running very very well for about 3.5 years now. > > > > Would lustre have issues if using cheap off the shell components or > > would people here think you need to have high end maskines with built in > > redundancy for everything? >---------------------------------------------------------------------------- Troy Benjegerdes ''da hozer'' hozer at hozed.org CTO, Freedom Fertilizer, Sustainable wind to NH3, troy at freedomfertilizer.com Benjegerdes Farms TerraCarbo biofuels The challenge in changing the world is not in having great ideas, it''s in having stupid simple ideas, as those are the ones that cause change. Intellectual property is one of those great complicated ideas that intellectuals like to intellectualize over, lawyers like to bill too much over, and engineers like to overengineer. Meanwhile, it''s the stupid simple ideas underfoot that create wealth. -- Troy, Mar 2010
Our success is based on simplicity. Software raid on direct attached disks with no add-on cards (ie. ensure MB''s have intel pro 1000 nics, at least 6 sata ports and reliable cpu''s etc). Our first generation gear consisted of a super-micro MB, 2GB memory single dual core intel cpu''s and 6x750GB direct attached disks in a white-box chassis running software raid 5. That was over 3.5years ago and it will actually be decommissioned tomorrow. 2nd generation were the same boxes, just the latest super-micro MB. 3rd generation were SGI xe250''s with 8x1TB direct attached disks with software raid5. 4th generation are SGI/Rackable systems with 12x2TB disks with an LSI/3ware hardware raid6 card. We absolutely hammer our file system and it has stood the test of time. I think our latest gear went in for ~$420/TB. -- Dr Stuart Midgley sdm900 at gmail.com On 23/04/2010, at 23:17 , Troy Benjegerdes wrote:> Taking a break from my current non-computer related work.. > > My guess based on your success is your gear is not so much cheap, as > *cost effective high MTBF commodity parts*. > > If you go for the absolute bargain basement stuff, you''ll have problems > as individual components flake out. > > If you spend way too much money on high-end multi-redundant whizbangs, > you generally get two things.. redundancy, which in my mind often only > serves to make the eventual failure worse, and high-quality, long MTBF > components. > > If you can get the high MTBF components without all the redudancy > (and associated complexity nightmare), then you win. > > > On Fri, Apr 23, 2010 at 05:42:30PM +0800, Stu Midgley wrote: >> We run lustre on cheap off the shelf gear. We have 4 generations of >> cheapish gear in a single 300TB lustre config (40 oss''s) >> >> It has been running very very well for about 3.5 years now. >> >> >>> Would lustre have issues if using cheap off the shell components or >>> would people here think you need to have high end maskines with built in >>> redundancy for everything?
Yes, we suffer hardware failures. All the time. That is sort of the point of Lustre and a clustered file system :) We have had double-disk failures with raid5 (recovered everything except ~1MB of data), server failures, MDS failures etc. We successfully recovered from them all. Sure, it can be a little stressful... but it all works. If server hardware fails, our file systems basically hangs until we fix it. Our most common failure is obviously disks... and they are all covered by raid. Since we have mostly direct attached disk, you have a few minutes downtime of a server while you replace the disk... everything continues as normal when the server comes back. -- Dr Stuart Midgley sdm900 at gmail.com On 23/04/2010, at 18:41 , Janne Aho wrote:> On 23/04/10 11:42, Stu Midgley wrote: > >>> Would lustre have issues if using cheap off the shell components or >>> would people here think you need to have high end machines with built in >>> redundancy for everything? >> >> We run lustre on cheap off the shelf gear. We have 4 generations of >> cheapish gear in a single 300TB lustre config (40 oss''s) >> >> It has been running very very well for about 3.5 years now. > > This sounds promising. > > Have you had any hardware failures? > If yes, how well has the cluster cooped with the loss of the machine(s)? > > > Any advice you can share from your initial setup of lustre?
Speaking of the future. Is there any more news about SNS? I think thats the only thing Lustre is missing to make it "production" ready and not just for research labs. On Fri, Apr 23, 2010 at 12:07 PM, Stuart Midgley <sdm900 at gmail.com> wrote:> Yes, we suffer hardware failures. ?All the time. ?That is sort of the point of Lustre and a clustered file system :) > > We have had double-disk failures with raid5 (recovered everything except ~1MB of data), server failures, MDS failures etc. ?We successfully recovered from them all. ?Sure, it can be a little stressful... but it all works. > > If server hardware fails, our file systems basically hangs until we fix it. ?Our most common failure is obviously disks... and they are all covered by raid. ?Since we have mostly direct attached disk, you have a few minutes downtime of a server while you replace the disk... everything continues as normal when the server comes back. > > -- > Dr Stuart Midgley > sdm900 at gmail.com > > > > On 23/04/2010, at 18:41 , Janne Aho wrote: > >> On 23/04/10 11:42, Stu Midgley wrote: >> >>>> Would lustre have issues if using cheap off the shell components or >>>> would people here think you need to have high end machines with built in >>>> redundancy for everything? >>> >>> We run lustre on cheap off the shelf gear. ?We have 4 generations of >>> cheapish gear in a single 300TB lustre config (40 oss''s) >>> >>> It has been running very very well for about 3.5 years now. >> >> This sounds promising. >> >> Have you had any hardware failures? >> If yes, how well has the cluster cooped with the loss of the machine(s)? >> >> >> Any advice you can share from your initial setup of lustre? > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
On 2010-04-26, at 05:29, Mag Gam wrote:> Speaking of the future. Is there any more news about SNS? I think > thats the only thing Lustre is missing to make it "production" ready > and not just for research labs.I agree, and this is one of the features that I will be advocating for our next round of development. That said, there were a lot of features promised in the past, with unrealistic expectations, so we are not going to be discussing/promoting "future" features that are possible to implement, so much as promoting features which have nearly finished implementation. Having input from users is definitely still useful in determining what features we will work on, so thanks for bringing this up.> On Fri, Apr 23, 2010 at 12:07 PM, Stuart Midgley <sdm900 at gmail.com> wrote: >> Yes, we suffer hardware failures. All the time. That is sort of the point of Lustre and a clustered file system :) >> >> We have had double-disk failures with raid5 (recovered everything except ~1MB of data), server failures, MDS failures etc. We successfully recovered from them all. Sure, it can be a little stressful... but it all works. >> >> If server hardware fails, our file systems basically hangs until we fix it. Our most common failure is obviously disks... and they are all covered by raid. Since we have mostly direct attached disk, you have a few minutes downtime of a server while you replace the disk... everything continues as normal when the server comes back. >> >> -- >> Dr Stuart Midgley >> sdm900 at gmail.com >> >> >> >> On 23/04/2010, at 18:41 , Janne Aho wrote: >> >>> On 23/04/10 11:42, Stu Midgley wrote: >>> >>>>> Would lustre have issues if using cheap off the shell components or >>>>> would people here think you need to have high end machines with built in >>>>> redundancy for everything? >>>> >>>> We run lustre on cheap off the shelf gear. We have 4 generations of >>>> cheapish gear in a single 300TB lustre config (40 oss''s) >>>> >>>> It has been running very very well for about 3.5 years now. >>> >>> This sounds promising. >>> >>> Have you had any hardware failures? >>> If yes, how well has the cluster cooped with the loss of the machine(s)? >>> >>> >>> Any advice you can share from your initial setup of lustre? >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discussCheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc.
Thought I would say thanks for all the input you have given and I''m sorry for misspelling of LustreFS. As we are completely green when it comes to any form of cluster file systems and we have to concider all the facts before we definitely know what we should do and how much time we have to calculate to build up a system, so that we can see if we can manage it "by ourselves" of if we would need to get in external help. Do you dare to estimate how long it would take to setup and tune a system with offers 40T storage. The main purpose is to use the storage to store VM-images which are used by KVM. Lest us assume that we use hardware RAID6. Today there will be ~20 clients using the storage. I know it''s a really difficult question to answer, specially when not saying how many machines there will be and so on (frankly we don''t know how it will be in the end), but I''m not really looking for how many minutes it will take, more roughly if it will be something that may take some days or quite many weeks or months... Thanks in advance for any replays. -- Janne Aho (Developer) | City Network Hosting AB - www.citynetwork.se Phone: +46 455 690022 | Cell: +46 733 312775 EMail/MSN: janne at citynetwork.se ICQ: 567311547 | Skype: janne_mz | AIM: janne4cn | Gadu: 16275665
At most a day. Most of that day will be inserting the DVD''s with Centos into the machines and installing :) The mkfs and mount of lustre will take a total of about an hour. -- Dr Stuart Midgley sdm900 at gmail.com On 28/04/2010, at 21:03 , Janne Aho wrote:> Thought I would say thanks for all the input you have given and I''m > sorry for misspelling of LustreFS. > > As we are completely green when it comes to any form of cluster file > systems and we have to concider all the facts before we definitely know > what we should do and how much time we have to calculate to build up a > system, so that we can see if we can manage it "by ourselves" of if we > would need to get in external help. > > Do you dare to estimate how long it would take to setup and tune a > system with offers 40T storage. The main purpose is to use the storage > to store VM-images which are used by KVM. Lest us assume that we use > hardware RAID6. Today there will be ~20 clients using the storage. > > I know it''s a really difficult question to answer, specially when not > saying how many machines there will be and so on (frankly we don''t know > how it will be in the end), but I''m not really looking for how many > minutes it will take, more roughly if it will be something that may take > some days or quite many weeks or months... > > > Thanks in advance for any replays. > > -- > Janne Aho (Developer) | City Network Hosting AB - www.citynetwork.se > Phone: +46 455 690022 | Cell: +46 733 312775 > EMail/MSN: janne at citynetwork.se > ICQ: 567311547 | Skype: janne_mz | AIM: janne4cn | Gadu: 16275665 > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
Your mention of hosting VM''s below: "to store VM-images which are used by KVM" is interesting. You''d probably benefit from some sort of de-duplication, which lustre doesn''t do. The workload would also seem to not fit into lustre''s key strengths. Have you considered using something like a ZFS backend, sharing out either iSCSI LUN''s or NFS to your 20 clients? On the surface, this looks like it would be a better fit. Have a look at the Oracle 7000 appliances if you want something turnkey, or look into one of the many ZFS appliances built on OpenSolaris...or build your own. $.02 -frank On Apr 28, 2010, at 8:26 AM, Stuart Midgley wrote:> At most a day. Most of that day will be inserting the DVD''s with Centos into the machines and installing :)Surely, you''ve heard of kickstart (or some other provisioning mechanism).> > The mkfs and mount of lustre will take a total of about an hour. > > > -- > Dr Stuart Midgley > sdm900 at gmail.com > > > > On 28/04/2010, at 21:03 , Janne Aho wrote: > >> Thought I would say thanks for all the input you have given and I''m >> sorry for misspelling of LustreFS. >> >> As we are completely green when it comes to any form of cluster file >> systems and we have to concider all the facts before we definitely know >> what we should do and how much time we have to calculate to build up a >> system, so that we can see if we can manage it "by ourselves" of if we >> would need to get in external help. >> >> Do you dare to estimate how long it would take to setup and tune a >> system with offers 40T storage. The main purpose is to use the storage >> to store VM-images which are used by KVM. Lest us assume that we use >> hardware RAID6. Today there will be ~20 clients using the storage. >> >> I know it''s a really difficult question to answer, specially when not >> saying how many machines there will be and so on (frankly we don''t know >> how it will be in the end), but I''m not really looking for how many >> minutes it will take, more roughly if it will be something that may take >> some days or quite many weeks or months... >> >> >> Thanks in advance for any replays. >> >> -- >> Janne Aho (Developer) | City Network Hosting AB - www.citynetwork.se >> Phone: +46 455 690022 | Cell: +46 733 312775 >> EMail/MSN: janne at citynetwork.se >> ICQ: 567311547 | Skype: janne_mz | AIM: janne4cn | Gadu: 16275665 >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss