Lustre discuss - Dec 2009 - New to lustre - Help setting up Lustre

Hi Guys,

I am working on building a small cluster sharing the lustre file system.  I
will be using an infiniband interconnect for all the compute/head nodes.
Can someone please assist me in figuring out which RPM''s should I
install on
my MDS & OSS.:

kernel-ib-1.4.2-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm
kernel-ib-1.4.2-2.6.18_128.7.1.el5.x86_64.rpm
kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1.x86_64.rpm
lustre-1.8.1.1-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm
lustre-client-1.8.1.1-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm
lustre-client-modules-1.8.1.1-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm
lustre-ldiskfs-3.0.9-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm
lustre-modules-1.8.1.1-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm

Should I be installing the kernel-ib* rpm?  I am planning on installing the
drivers that the vendor provides for inifiniband.  I am pretty new to this
whole setup, so if any of this sounds too basic, my apologies in advance.

Also, what is the recommendation on using LVM for MDT''s and
OST''s?  Should
it be avoided or there is not much of a performance hit if we use LVM.

And finally, is there any reasons why I should not use large size luns
(approx. 2TB) for my OST''s?

Thanks,
-J
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091223/c9caf2e9/attachment.html

On Wed, 2009-12-23 at 14:44 -0800, Jagga Soorma wrote:
> Hi Guys,
Hi,
> which RPM''s should I install on my MDS & OSS.:
> 
> kernel-ib-1.4.2-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm
> kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1.x86_64.rpm
> lustre-1.8.1.1-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm
> lustre-ldiskfs-3.0.9-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm
> lustre-modules-1.8.1.1-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm
As well, you should install the e2fsprogs package for your distro and
architecture.
> Should I be installing the kernel-ib* rpm?
If you want OFED, yes.
> I am planning on installing the drivers that the vendor provides for
> inifiniband.
This will not work.  You have to use OFED modules that were built with
the kernel you are installing and you have to use Lustre modules that
were built with the OFED modules for your kernel.  I highly recommend
using what we provide for simplicity.

Once you have your feet wet you might have better success in deviating
from the straightforward path.
> Also, what is the recommendation on using LVM for MDT''s and
OST''s?
That should be fine.  I do that here at home.  We have a number of
clients that do this.  It facilitates backups of the MDT and OSTs
without having to take them down, should you want to do backups at some
point.
> Should it be avoided or there is not much of a performance hit if we
> use LVM.
I''ve never been convinced that there is a significant performance hit.
I have personally been using LVM as my exclusive block device
provisioning tool for a dozen years or more.  YMMV however.  If you want
to have scientific results before you start, simply benchmark a raw
device and then create an LV and benchmark it.  Report here your
differences just for interest''s sake.
> And finally, is there any reasons why I should not use large size luns
> (approx. 2TB) for my OST''s?
No.  We encourage the use of fewer and larger (rather than more and
smaller) LUNS, to the limit of the (current) 8TB max for LUNs.  That
limit gets higher in 1.8.2 IIRC.

Also, there is really no point in creating more than one OST (i.e. LUN)
per "spindle" if that''s an option for you.  This is not
usually a
problem for most people as most people are creating LUNs larger than the
largest single disks you can commonly buy anyway.

Cheers and good luck in your Lustre adventure.

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091223/31267111/attachment.bin

Hi,

On Dec 23, 2009, at 3:08 PM, Brian J. Murrell wrote:
>> Should it be avoided or there is not much of a performance hit if we
>> use LVM.
>
> I''ve never been convinced that there is a significant performance
hit.
> I have personally been using LVM as my exclusive block device
> provisioning tool for a dozen years or more.  YMMV however.  If you  
> want
> to have scientific results before you start, simply benchmark a raw
> device and then create an LV and benchmark it.  Report here your
> differences just for interest''s sake.
We just put one large (860TB), and are about to put two more 430TB  
filesystems into production using LVM. Backend disk for the OSTs is  
7200RPM 1TB SATA on DDN 9900 controllers, and for the MDT is 15000RPM  
400GB SAS on an SGI IS200 in a RAID 10. Performance *without*  
snapshots seemed to be fine. However, with snapshots in play on the  
OSTs, performance is about 1/10th of what we normally see, due to a  
mix of small (8KB-64KB) reads and writes when performing copy-on-write  
of blocks written when a snapshot is active. IIRC, the 8K I/Os were  
with the default snapshot chunk size, and the 64KB I/Os were with a  
512KB chunk size.

We decided to do snapshots for two reasons - live backups of the MDT,  
and to perform live fscks on the MDT and OSTs if we suspect data  
corruption. For those limited uses, we''re willing to occasionally take
the performance hit. I wouldn''t do this to regularly back up the OSTs,
though, which is not something we do with these large scratch  
filesystems.

YMMV. As Brian said, nothing beats doing the testing yourself to  
understand how your hardware behaves, for both benchmarks and, more  
importantly, the applications you run.

j

--
Jason Rappleye
System Administrator
NASA Advanced Supercomputing Division
NASA Ames Research Center
Moffett Field, CA 94035
jason.rappleye at nasa.gov

>As well, you should install the e2fsprogs package for your distro and
>architecture.
As long as we''re talking about this ... in the case of SLES 11, you
need to install some extra, hidden RPMs to get e2fsprogs to work (I
found those by accident on downloads.lustre.org after spending a bunch
of time trying to figure out what was wrong with e2fsprogs).  Could
these be added to the to the normal download site?

--Ken

Ken

Did you open a bug to report this?

PJones

Ken Hornstein wrote:
>> As well, you should install the e2fsprogs package for your distro and
>> architecture.
>>     
>
> As long as we''re talking about this ... in the case of SLES 11,
you
> need to install some extra, hidden RPMs to get e2fsprogs to work (I
> found those by accident on downloads.lustre.org after spending a bunch
> of time trying to figure out what was wrong with e2fsprogs).  Could
> these be added to the to the normal download site?
>
> --Ken
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>

>Did you open a bug to report this?
No, I will confess that I did not.  I did post an email a while back about
the original problem I was having, and I got no replies ... I dug further,
found the hidden RPMs, and I was kind of under a deadline at the time, so
the followup sort of slipped my mind.  Brian''s post about e2fsprogs
jogged
my memory.

If you want me to do that, I''ll be glad to.

--Ken

Hello Ken,
	I think the original problem you were referring to is "?undefined
symbol: ext2_attr_index_prefix " ? This is fixed in bug#20830 and
checked in to the cvs. The fix will be available with release of
e2fsprogs 1.41.9-sun1.

Thanks,
Girish

On Mon, 2009-12-28 at 19:36 -0500, Ken Hornsten wrote:
> >Did you open a bug to report this?
> 
> No, I will confess that I did not.  I did post an email a while back about
> the original problem I was having, and I got no replies ... I dug further,
> found the hidden RPMs, and I was kind of under a deadline at the time, so
> the followup sort of slipped my mind.  Brian''s post about
e2fsprogs jogged
> my memory.
> 
> If you want me to do that, I''ll be glad to.
> 
> --Ken
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

Hi Ken

 From Girish''s post it seems that this is a known issue that has now 
been fixed, but, in general, yes, opening a ticket is always the most 
reliable way of making sure that we track and fix and given problem.

Thanks

PJones

Ken Hornsten wrote:
>> Did you open a bug to report this?
>>     
>
> No, I will confess that I did not.  I did post an email a while back about
> the original problem I was having, and I got no replies ... I dug further,
> found the hidden RPMs, and I was kind of under a deadline at the time, so
> the followup sort of slipped my mind.  Brian''s post about
e2fsprogs jogged
> my memory.
>
> If you want me to do that, I''ll be glad to.
>
> --Ken
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>

>	I think the original problem you were referring to is "???undefined
>symbol: ext2_attr_index_prefix " ? This is fixed in bug#20830 and
>checked in to the cvs. The fix will be available with release of
>e2fsprogs 1.41.9-sun1.
Actually, I just looked at this bug, and this is not the problem (the issue
I had was actually a different symbol, and it had mmp in the name of the
symbol).

What I was referring to is that if you go to the Sun web site to download
Lustre and you select SLES 11, one of the RPMs listed is the one for
e2fsprogs.  That''s fine.  But what it DOESN''T list is these
RPMs:

http://downloads.lustre.org/public/tools/e2fsprogs/1.41.6.sun1/sles11/x86_64/

Specifically, you need all of those library RPMs to make e2fsprogs work
under SLES 11.  This really isn''t mentioned anywhere that I saw.  So
it''s not a BUG in Lustre, per se ... it''s really more of a
packaging
issue on the Sun download web site.  If that''s still the sort of thing
that
should go in a bug report, I''ll be glad to do it.

--Ken

Ah, I see. I was actually referring to subject line of mail, you sent on
9th October. Anyways, regarding the packaging issue I will check with
our release manager to resolve this.

Regards,
-Girish

On Wed, 2009-12-30 at 10:09 -0500, Ken Hornstein wrote:
> >	I think the original problem you were referring to is "?undefined
> >symbol: ext2_attr_index_prefix " ? This is fixed in bug#20830 and
> >checked in to the cvs. The fix will be available with release of
> >e2fsprogs 1.41.9-sun1.
> 
> Actually, I just looked at this bug, and this is not the problem (the issue
> I had was actually a different symbol, and it had mmp in the name of the
> symbol).
> 
> What I was referring to is that if you go to the Sun web site to download
> Lustre and you select SLES 11, one of the RPMs listed is the one for
> e2fsprogs.  That''s fine.  But what it DOESN''T list is
these RPMs:
> 
>
http://downloads.lustre.org/public/tools/e2fsprogs/1.41.6.sun1/sles11/x86_64/
> 
> Specifically, you need all of those library RPMs to make e2fsprogs work
> under SLES 11.  This really isn''t mentioned anywhere that I saw. 
So
> it''s not a BUG in Lustre, per se ... it''s really more of
a packaging
> issue on the Sun download web site.  If that''s still the sort of
thing that
> should go in a bug report, I''ll be glad to do it.
> 
> --Ken