Hi, I have just recently started to test Lustre and even with a small 2 nodes deployment, I find the performance is outstanding on 1Gb networking. I was trying to search through the list for anything that could help me to mount using NFS protocol but could not find anything. Is that possible or have anyone done that before? I am in particular interested if anyone have use Lustre with VMware. I am currently testing on 2.1.5. Thanks in advance. Regards Beh _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Hi Beh, I have no experience with Lustre and NFS, but this past spring (are we out of it yet?) ;) I had the chance to learn a lot about Lustre as I chose it to be the subject of a term paper for the college I am attending. I decided an hands on approach was the best way and started out with 3 nodes and tried to get things running. Being inexperienced with this wonderful system, I get blowing things up and running my install of CentOS. I decided to purchase a server big enough to support a 6 node Lustre cluster in VMware ESXi 5.0 update 2. I have each client, the OSS, and MDT running on their own drives on a Gb network. I am astonished with the performance that I am seeing with this setup and am very pleased with how it runs. I am going to add some physical machines to this setup here in the near future when I have a few more minutes (I have to pay for college somehow) and then allocate the resources from the old 4 VM clients to the MDT and OSS and see what performance increases I see that way. I wished I had a SAN with fiber connections to squeeze out all of the performance I could, but one day I will try it! :) If you have any specific questions in regards to VMware I would be more than happy to answer them to the best of my abilities. Once again, it runs beautifully and I haven''t seen any errors or problems since inception on this system. I am also running CentOS 6.4 with Lustre 2.4 and collectl for some basic statistics here and there. Hope this helps in some way or another! Trevor Hargraves On Thu, Jun 20, 2013 at 9:57 PM, Teik Hooi Beh <thbeh-tsXiyn/lwR0AvxtiuMwx3w@public.gmane.org> wrote:> Hi, > > I have just recently started to test Lustre and even with a small 2 nodes > deployment, I find the performance is outstanding on 1Gb networking. > > I was trying to search through the list for anything that could help me to > mount using NFS protocol but could not find anything. > > Is that possible or have anyone done that before? I am in particular > interested if anyone have use Lustre with VMware. I am currently testing on > 2.1.5. > > Thanks in advance. > > Regards > Beh > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >-- Thanks, Trevor Hargraves _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Hi Trevor, Thanks for the well define explanation. I see that you have a successful testing with Lustre cluster in ESXi. In my case I was more of hoping to use Lustre as an alternate storage for ESXi instead of expensive SAN. While I am aware and have a couple of successful deployment of ESX with external NFS storage (VMWare allows to mount NFS store from within vCenter), but I am not able to mount Lustre storage. I see this as an excuse to buy expensive SAN to increase storage capacity. Regards Beh On Fri, Jun 21, 2013 at 12:42 PM, Trevor Hargraves <hargtre2-1eGbkeHAdCU@public.gmane.org> wrote:> Hi Beh, > > I have no experience with Lustre and NFS, but this past spring (are we out > of it yet?) ;) I had the chance to learn a lot about Lustre as I chose it > to be the subject of a term paper for the college I am attending. I decided > an hands on approach was the best way and started out with 3 nodes and > tried to get things running. Being inexperienced with this wonderful > system, I get blowing things up and running my install of CentOS. I decided > to purchase a server big enough to support a 6 node Lustre cluster in > VMware ESXi 5.0 update 2. I have each client, the OSS, and MDT running on > their own drives on a Gb network. I am astonished with the performance that > I am seeing with this setup and am very pleased with how it runs. I am > going to add some physical machines to this setup here in the near future > when I have a few more minutes (I have to pay for college somehow) and then > allocate the resources from the old 4 VM clients to the MDT and OSS and see > what performance increases I see that way. I wished I had a SAN with fiber > connections to squeeze out all of the performance I could, but one day I > will try it! :) > > If you have any specific questions in regards to VMware I would be more > than happy to answer them to the best of my abilities. Once again, it runs > beautifully and I haven''t seen any errors or problems since inception on > this system. I am also running CentOS 6.4 with Lustre 2.4 and collectl for > some basic statistics here and there. > > > Hope this helps in some way or another! > > Trevor Hargraves > > > On Thu, Jun 20, 2013 at 9:57 PM, Teik Hooi Beh <thbeh-tsXiyn/lwR0AvxtiuMwx3w@public.gmane.org> wrote: > >> Hi, >> >> I have just recently started to test Lustre and even with a small 2 nodes >> deployment, I find the performance is outstanding on 1Gb networking. >> >> I was trying to search through the list for anything that could help me >> to mount using NFS protocol but could not find anything. >> >> Is that possible or have anyone done that before? I am in particular >> interested if anyone have use Lustre with VMware. I am currently testing on >> 2.1.5. >> >> Thanks in advance. >> >> Regards >> Beh >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> >> > > > -- > > Thanks, > > Trevor Hargraves >_______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Lustre files can be re-exported using NFS .>From the presentation by Oleg in Lug-2008 (http://wiki.lustre.org/images/d/da/LUG08-Lustre-NFS.pdf ) Lustre is a normal local fs from NFS perspective Just add an entry to /etc/export *on lustre client*/NFS server give it try. Hope that helps. parinay On 21 June 2013 09:27, Teik Hooi Beh <thbeh-tsXiyn/lwR0AvxtiuMwx3w@public.gmane.org> wrote:> Hi, > > I have just recently started to test Lustre and even with a small 2 nodes > deployment, I find the performance is outstanding on 1Gb networking. > > I was trying to search through the list for anything that could help me to > mount using NFS protocol but could not find anything. > > Is that possible or have anyone done that before? I am in particular > interested if anyone have use Lustre with VMware. I am currently testing on > 2.1.5. > > Thanks in advance. > > Regards > Beh > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >_______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Thanks for the pointer. Just a quick question, will the performance suffer since we have to add another NFS layer to the lustre client? Regards Beh On Fri, Jun 21, 2013 at 4:07 PM, Parinay Kondekar < parinay_kondekar-qCPWdT176rRBDgjK7y7TUQ@public.gmane.org> wrote:> Lustre files can be re-exported using NFS . > > From the presentation by Oleg in Lug-2008 ( > http://wiki.lustre.org/images/d/da/LUG08-Lustre-NFS.pdf ) > > Lustre is a normal local fs from NFS perspective Just add an entry to > /etc/export *on lustre client*/NFS server > > give it try. > > Hope that helps. > parinay > > > On 21 June 2013 09:27, Teik Hooi Beh <thbeh-tsXiyn/lwR0AvxtiuMwx3w@public.gmane.org> wrote: > >> Hi, >> >> I have just recently started to test Lustre and even with a small 2 nodes >> deployment, I find the performance is outstanding on 1Gb networking. >> >> I was trying to search through the list for anything that could help me >> to mount using NFS protocol but could not find anything. >> >> Is that possible or have anyone done that before? I am in particular >> interested if anyone have use Lustre with VMware. I am currently testing on >> 2.1.5. >> >> Thanks in advance. >> >> Regards >> Beh >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> >> >_______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
From: Teik Hooi Beh <thbeh-tsXiyn/lwR0AvxtiuMwx3w@public.gmane.org<mailto:thbeh-tsXiyn/lwR0AvxtiuMwx3w@public.gmane.org>> Date: Friday, June 21, 2013 1:29 AM To: Parinay Kondekar <parinay_kondekar-qCPWdT176rRBDgjK7y7TUQ@public.gmane.org<mailto:parinay_kondekar-qCPWdT176rRBDgjK7y7TUQ@public.gmane.org>> Cc: "lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org>" <lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org>> Subject: Re: [Lustre-discuss] Using NFS to mount lustre Thanks for the pointer. Just a quick question, will the performance suffer since we have to add another NFS layer to the lustre client? To be clear, you cannot export Lustre via NFS from Lustre servers. You re-export the Lustre filesystem from a Lustre client node. Essentially, you turn one client into an NFS server. So, the bottleneck in performance is that all NFS traffic must come from that node, and you are limited to the single-node network bandwidth, instead of streaming parallel IO from all servers with the native Lustre client. In a small setup such as you describe you will likely not see any performance impact. Cliffw Regards Beh On Fri, Jun 21, 2013 at 4:07 PM, Parinay Kondekar <parinay_kondekar-qCPWdT176rRBDgjK7y7TUQ@public.gmane.org<mailto:parinay_kondekar-qCPWdT176rRBDgjK7y7TUQ@public.gmane.org>> wrote: Lustre files can be re-exported using NFS .>From the presentation by Oleg in Lug-2008 ( http://wiki.lustre.org/images/d/da/LUG08-Lustre-NFS.pdf )Lustre is a normal local fs from NFS perspective Just add an entry to /etc/export on lustre client/NFS server give it try. Hope that helps. parinay On 21 June 2013 09:27, Teik Hooi Beh <thbeh-tsXiyn/lwR0AvxtiuMwx3w@public.gmane.org<mailto:thbeh-tsXiyn/lwR0AvxtiuMwx3w@public.gmane.org>> wrote: Hi, I have just recently started to test Lustre and even with a small 2 nodes deployment, I find the performance is outstanding on 1Gb networking. I was trying to search through the list for anything that could help me to mount using NFS protocol but could not find anything. Is that possible or have anyone done that before? I am in particular interested if anyone have use Lustre with VMware. I am currently testing on 2.1.5. Thanks in advance. Regards Beh _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> http://lists.lustre.org/mailman/listinfo/lustre-discuss
On Fri, Jun 21, 2013 at 7:51 PM, White, Cliff <cliff.white-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:> From: Teik Hooi Beh <thbeh-tsXiyn/lwR0AvxtiuMwx3w@public.gmane.org<mailto:thbeh-tsXiyn/lwR0AvxtiuMwx3w@public.gmane.org>> > Date: Friday, June 21, 2013 1:29 AM > To: Parinay Kondekar <parinay_kondekar-qCPWdT176rRBDgjK7y7TUQ@public.gmane.org<mailto:parinay_kondekar-qCPWdT176rRBDgjK7y7TUQ@public.gmane.org>> > Cc: "lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org>" <lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org>> > Subject: Re: [Lustre-discuss] Using NFS to mount lustre > > Thanks for the pointer. > > Just a quick question, will the performance suffer since we have to add another NFS layer to the lustre client? > > To be clear, you cannot export Lustre via NFS from Lustre servers. You re-export the Lustre filesystem from a Lustre client node. > Essentially, you turn one client into an NFS server. > So, the bottleneck in performance is that all NFS traffic must come from that node, and you are limited to the single-node network bandwidth, instead of streaming parallel IO from all servers with the native Lustre client. In a small setup such as you describe you will likely not see any performance impact. > CliffwI may be wrong, but would there be any difference what so ever for the OP between using a lustre as a backend and just setting up an NFS server on a *nix box with some storage attached? Regards, ELi> > > Regards > Beh > > > On Fri, Jun 21, 2013 at 4:07 PM, Parinay Kondekar <parinay_kondekar-qCPWdT176rRBDgjK7y7TUQ@public.gmane.org<mailto:parinay_kondekar-qCPWdT176rRBDgjK7y7TUQ@public.gmane.org>> wrote: > Lustre files can be re-exported using NFS . > > >From the presentation by Oleg in Lug-2008 ( http://wiki.lustre.org/images/d/da/LUG08-Lustre-NFS.pdf ) > > Lustre is a normal local fs from NFS perspective Just add an entry to /etc/export on lustre client/NFS server > > give it try. > > Hope that helps. > parinay > > > On 21 June 2013 09:27, Teik Hooi Beh <thbeh-tsXiyn/lwR0AvxtiuMwx3w@public.gmane.org<mailto:thbeh-tsXiyn/lwR0AvxtiuMwx3w@public.gmane.org>> wrote: > Hi, > > I have just recently started to test Lustre and even with a small 2 nodes deployment, I find the performance is outstanding on 1Gb networking. > > I was trying to search through the list for anything that could help me to mount using NFS protocol but could not find anything. > > Is that possible or have anyone done that before? I am in particular interested if anyone have use Lustre with VMware. I am currently testing on 2.1.5. > > Thanks in advance. > > Regards > Beh > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
Hi Rosenberg, On Sun, Jun 23, 2013 at 7:24 PM, E.S. Rosenberg <esr+lustre-ed8gSyzkC85vlA8e7MYcDA@public.gmane.org>wrote:> I may be wrong, but would there be any difference what so ever for the > OP between using a lustre as a backend and just setting up an NFS > server on a *nix box with some storage attached? > Regards, > ELi >There might be many reasons. For example, in a mixed environment where you need NFS access to the data on a Lustre file system and the client''s platform might be not supported by Lustre. Or there might be too large amount of data stored on the Lustre to make a transition. By the way, topic author may consider of using (or waiting for) a pNFS layout driver for Lustre to take the advantages of both NFS and parallel file system. http://datatracker.ietf.org/doc/draft-faibish-nfsv4-pnfs-lustre-layout/ http://www.nfsv4bat.org/Documents/ConnectAThon/2012/pNFS4Lustre-cthon-2012.pdf Regards, Cuong _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss