Lustre discuss - Jan 2008 - struggling just get started

I''d like to install Lustre on some hosts (and, yes, I''ve read
the new found issue with the current release with filesystems under 20000
files).  But I''m having a bit of a problem in determining just how to
build this thing.

I''m installing under SLES 10.  I''d like to run over the IB
interface.

The first item of business appears that I can download a kernel already built
for my machine.  Talk on the network seems to indicate that this will not run
over an IB interface unless I compile a new kernel from sources.  Yet I
don''t quite understand why this might be as I don''t have toi
recompile running over a gigE connection, regardless of the actual interface.
Whether it be an Intel or a Broadcom, it doesn''t appear to matter and I
don''t need to recompile against those sources.  Which makes me question
why I''d need to recompile when using IB.

It would seem that if I chose TCPoIB that the actual kernel drivers should
handle this just fine, running over a TCP stack and the actual device driver
should do "the right thing" by making these packets available over the
correct physical device.

Now maybe there is a performance issue in doing it this way.  and maybe it just
doesn''t work that way at all.

So I''ve read some things and downloaded the sources, both the modified
kernel and the lustre-source RPMs.  Now it want me to supply some link to the
openib source directory.  How do I determine which OFED was installed with the
kernel so I know which sources to download?  Aren''t these drivers a
part of the kernel now?  And if so, houldn''t they actually be already
there in the kernel sources?

Thanks for any help anyone can supply with helping me to understand how to even
start this build process.

Bill

On Sat, Jan 19, 2008 at 11:18:15AM -0500, William G Wichser
wrote:
>The first item of business appears that I can download a kernel already
built for my machine.  Talk on the network seems to indicate that this will not
run over an IB interface unless I compile a new kernel from sources.  Yet I
don''t quite understand why this might be as I don''t have toi
recompile running over a gigE connection, regardless of the actual interface.
Whether it be an Intel or a Broadcom, it doesn''t appear to matter and I
don''t need to recompile against those sources.  Which makes me question
why I''d need to recompile when using IB.
the ko2iblnd.ko Lustre kernel module needs to be built in order to use
the IB native protocols, and Sun doesn''t seem to do this at the moment.
I don''t know why. it''s not there for sles10 or for rhel5, I
haven''t
checked the rest.
>It would seem that if I chose TCPoIB that the actual kernel drivers should
handle this just fine, running over a TCP stack and the actual device driver
should do "the right thing" by making these packets available over the
correct physical device.
IPoIB will work, and it''ll be faster than GigE, but nowhere near as
fast
as using native IB (verbs etc.)
>Now maybe there is a performance issue in doing it this way.  and maybe it
just doesn''t work that way at all.
>So I''ve read some things and downloaded the sources, both the
modified kernel and the lustre-source RPMs.  Now it want me to supply some link
to the openib source directory.  How do I determine which OFED was installed
with the kernel so I know which sources to download?  Aren''t these
drivers a part of the kernel now?  And if so, houldn''t they actually be
already there in the kernel sources?
unless you have a specific reason not to then it''s simplest to just
point Lustre''s configure at the kernel''s IB implementation.
eg.
  ./configure --with-linux=<kpath> --with-o2ib=<kpath>
where <kpath> might be /usr/src/linux-2.6.5-7.199/ or
/lib/modules/2.6.5-7.199-sn2/build/ or something like that. you''ll need
the sles10 devel or source rpm or whatever it''s called installed to get
this tree.
>Thanks for any help anyone can supply with helping me to understand how to
even start this build process.
kinda depends if you want a patched server or patchless.

if patchless then just the above will suffice.
if patching then you need to patch (and preferably build) your kernel
tree prior to doing the above configure. I think the Sun recommended
way to patch is via quilt, or you can apply each of the patches in
  lustre/kernel_patches/series/2.6-sles10.series

IMHO the patched kernels are usually more stable than the patchless,
but you need to rebuild the kernel so they''re a tad more of a pain.

cheers,
robin