Lustre discuss - Jun 2012 - Lustre 2.x support on vanilla kernels

Hello,

we are building an HPC IB-backed cluster and are considering to
use Lustre 2.x for our distributed and parallel cluster storage.

What bothers me that only RedHat-based systems are claimed to be
supported. We will likely use Gentoo on our computing nodes, this
effectively means vanilla kernels. On Lustre servers we may install
RedHat, this is an acceptable, but not a desirable solution.

On clients we will definitely use 3.x kernels, so I''d like to know
a current status of Lustre client support. They are patchless to my
knowledge, so should work on any recent kernel, but your mileage may
vary.

Also if someone have an experience with Lustre 2.x servers on vanilla
kernels, it will be welcome.

Best regards,
Andrew Savchenko
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120605/e4294c42/attachment.bin

On 2012-06-05, at 7:11 AM, Andrew Savchenko wrote:
> we are building an HPC IB-backed cluster and are considering to
> use Lustre 2.x for our distributed and parallel cluster storage.
> 
> What bothers me that only RedHat-based systems are claimed to be
> supported. We will likely use Gentoo on our computing nodes, this
> effectively means vanilla kernels. On Lustre servers we may install
> RedHat, this is an acceptable, but not a desirable solution.
On servers, the reason that we only support vendor kernels is because
there are patches needed for ext4 to support extra features that we
have implemented.  That means it takes some non-trivial effort to add
support for a different kernel.  With Lustre 2.4 + ZFS it will be
possible to run Lustre servers completely without patches.

Since the Lustre servers are intended to be separate from the rest of
the compute cluster, it is not usually a problem to run RHEL on those
nodes, and whatever distro you want on the client nodes (assuming you
want to build your own packages for them).
> On clients we will definitely use 3.x kernels, so I''d like to know
> a current status of Lustre client support. They are patchless to my
> knowledge, so should work on any recent kernel, but your mileage may
> vary.
There are continuous changes to the kernel APIs that filesystems need
to use to work on Linux.  In 2.6.38 there was a major change to how
clients interact with the dcache, which will be included in Lustre
clients for the 2.3 release.  There are some patches for making the
Lustre client work with 3.1 and 3.3 kernels, but those patches are not
yet landed.

Having interested users test newer kernels and report back their
experience to the list helps everyone involved.  It allows users
such as yourself to participate in the development of Lustre, and
allows maintainers to have some assurance that the patches for those
kernels actually allow Lustre to work on the newer kernels, and not
just compile without errors.

As you can imagine, we cannot possibly build and test every combination
of Linux kernels and distros available.  We do include tools to build
packages for Debian/Ubuntu based systems, and do regular builds for
Ubuntu 10, but as yet there has been little interest in Gentoo.  If
this is something you feel strongly about, possibly you could contribute
Makefile rules/scripts to build Gentoo packages to help others also
use Lustre with Gentoo.
> Also if someone have an experience with Lustre 2.x servers on vanilla
> kernels, it will be welcome.
> 
> Best regards,
> Andrew Savchenko
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

Cheers, Andreas
--
Andreas Dilger                       Whamcloud, Inc.
Principal Lustre Engineer            http://www.whamcloud.com/




-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 235 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120605/a90bf39f/attachment.bin

Hello Andreas,

On Tue, 5 Jun 2012 09:16:51 -0600 Andreas Dilger wrote:
> On 2012-06-05, at 7:11 AM, Andrew Savchenko wrote:
> > we are building an HPC IB-backed cluster and are considering to
> > use Lustre 2.x for our distributed and parallel cluster storage.
> > 
> > What bothers me that only RedHat-based systems are claimed to be
> > supported. We will likely use Gentoo on our computing nodes, this
> > effectively means vanilla kernels. On Lustre servers we may install
> > RedHat, this is an acceptable, but not a desirable solution.
> 
> On servers, the reason that we only support vendor kernels is because
> there are patches needed for ext4 to support extra features that we
> have implemented.  That means it takes some non-trivial effort to add
> support for a different kernel.  With Lustre 2.4 + ZFS it will be
> possible to run Lustre servers completely without patches.
That''s a good news to hear. Is there any time schedule for future
Lustre releases? On lustre.org I can find only up to version 2.0.
 
> Since the Lustre servers are intended to be separate from the rest of
> the compute cluster, it is not usually a problem to run RHEL on those
> nodes, 
There may be some problems with running too old kernel on too new
hardware, but I hope we''ll find a fay to handle this.
> and whatever distro you want on the client nodes (assuming you
> want to build your own packages for them).
It shouldn''t be a problem if it will not require a heavy kernel code
patching.
> > On clients we will definitely use 3.x kernels, so I''d like to
know
> > a current status of Lustre client support. They are patchless to my
> > knowledge, so should work on any recent kernel, but your mileage may
> > vary.
> 
> There are continuous changes to the kernel APIs that filesystems need
> to use to work on Linux.  In 2.6.38 there was a major change to how
> clients interact with the dcache, which will be included in Lustre
> clients for the 2.3 release.  There are some patches for making the
> Lustre client work with 3.1 and 3.3 kernels, but those patches are not
> yet landed.
> 
> Having interested users test newer kernels and report back their
> experience to the list helps everyone involved.  It allows users
> such as yourself to participate in the development of Lustre, and
> allows maintainers to have some assurance that the patches for those
> kernels actually allow Lustre to work on the newer kernels, and not
> just compile without errors.
> 
> As you can imagine, we cannot possibly build and test every combination
> of Linux kernels and distros available.  We do include tools to build
> packages for Debian/Ubuntu based systems, and do regular builds for
> Ubuntu 10, but as yet there has been little interest in Gentoo.  If
> this is something you feel strongly about, possibly you could contribute
> Makefile rules/scripts to build Gentoo packages to help others also
> use Lustre with Gentoo.
 
Okay, let''s start from a Gentoo client. I have some experience in
package creation and support, so this shouldn''t be that hard if it
comes to build subsystem and packaging. But I''m new to Lustre and if
it will come to serious code changes, I will need a help from
developers.

But at this very moment I have troubles with testing grounds. Our
cluster is in the early construction stage now, so I will not be able
to test a package on the actual hardware. Estimated completion time
to the point when I''ll be able to test Lustre is 6 +5/-3 months. I
can setup some virtual test environment, but it will have a poor
interconnect (no more than 1 Gb/s ethernet) and a few hosts, so
emulation may be inadequate.

Anyway I need some time to study Lustre docs in order to understand
its internals and how to set up it properly. When I''ll have a
prepared test environment I''ll let you know (this may take some
considerable time as I have other duties).

As for now, should I stick to something like Lustre-2.3 branch,
RHEL-5.4 on the server and 2.6.38-3.0 kernels on the clients?

Best regards,
Andrew Savchenko
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120606/d0804006/attachment.bin

Hello Mark,

On Tue, 5 Jun 2012 17:53:15 -0400 (EDT) Mark Hahn wrote:
> > What bothers me that only RedHat-based systems are claimed to be
> 
> is it safe to assume you know this includes Centos?
Yes, of course. And Scientific Linux too. But currently supported
kernels are RH-5.4-based (for the latest released Lustre 2.0 series).
I doubt Lustre will work without serious changes on RHEL-6.x
versions and related distributions.
> > supported. We will likely use Gentoo on our computing nodes, this
> 
> out of curiosity, why?
There are numerous reasons. In short they are: extreme flexibility
and convenient process of deep updates. And for some applications
performance benefits from careful optimization are really good, but
this reason is secondary.

In more details, I have an experience with binary distributions on
the clusters, mostly RH-based like SLC. While they are easy to
deploy, they are hard in maintenance for our work profile.

We do not have a small, limited number of supported applications, we
need to allow users to use numerous applications, thus we need to
install numerous libraries and packages. Many if not most of them are
absent in the main distribution repository and third-party
repositories are not binary compatible with each other. This creates
a pain, a lot of pain; in the end it is needed to supported own
repository with packaged build from the source code. And where it
comes to building a large amount of packages from the source, Gentoo
rocks, because this is what it were created for.

Also Gentoo is extremely flexible in fine-tuned system configuration,
it is easy to remove unneeded components, which improves security and
simplifies maintenance (you can''t do that in binary distributions:
e.g. if program you need links with libABC, requiring daemon ABC to
be running, and you do not need it, you can''t just remove it due to
the linking dependencies).

Yet another benefit is a rolling update: it is possible to gradually
update even several years old system without re-installation from
scratch with convenient and half-automated config updates. Of course,
there are ways for a gradual update of a binary distribution, e.g.
RHEL-5.0 to RHEL-6.1, but they are too painful to my taste and often
are not supported by the distribution at all (you''re on your own).

There is a drawback of course: it takes longer to perform initial
fine-tuned system setup. As usual, one needs to pay for benefits by
some overhead in other areas.

Of course, you mileage may vary depending on your needs. If you are
happy with your current setup, there are no reasons to change things.

Best regards,
Andrew Savchenko
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120606/fa7e2147/attachment.bin

On 2012-06-05, at 4:47 PM, Andrew Savchenko wrote:
> On Tue, 5 Jun 2012 17:53:15 -0400 (EDT) Mark Hahn wrote:
>>> What bothers me that only RedHat-based systems are claimed to be
>> 
>> is it safe to assume you know this includes Centos?
> 
> Yes, of course. And Scientific Linux too. But currently supported
> kernels are RH-5.4-based (for the latest released Lustre 2.0 series).
> I doubt Lustre will work without serious changes on RHEL-6.x
> versions and related distributions.
It is important to note that Whamcloud is making Lustre 1.8.x releases with
RHEL6 client support, and 2.1.x releases with RHEL6 server support.  These
releases are not on the lustre.org website, but rather from
http://www.whamcloud.com/downloads/.

Cheers, Andreas
>>> supported. We will likely use Gentoo on our computing nodes, this
>> 
>> out of curiosity, why?
> 
> There are numerous reasons. In short they are: extreme flexibility
> and convenient process of deep updates. And for some applications
> performance benefits from careful optimization are really good, but
> this reason is secondary.
> 
> In more details, I have an experience with binary distributions on
> the clusters, mostly RH-based like SLC. While they are easy to
> deploy, they are hard in maintenance for our work profile.
> 
> We do not have a small, limited number of supported applications, we
> need to allow users to use numerous applications, thus we need to
> install numerous libraries and packages. Many if not most of them are
> absent in the main distribution repository and third-party
> repositories are not binary compatible with each other. This creates
> a pain, a lot of pain; in the end it is needed to supported own
> repository with packaged build from the source code. And where it
> comes to building a large amount of packages from the source, Gentoo
> rocks, because this is what it were created for.
> 
> Also Gentoo is extremely flexible in fine-tuned system configuration,
> it is easy to remove unneeded components, which improves security and
> simplifies maintenance (you can''t do that in binary distributions:
> e.g. if program you need links with libABC, requiring daemon ABC to
> be running, and you do not need it, you can''t just remove it due
to
> the linking dependencies).
> 
> Yet another benefit is a rolling update: it is possible to gradually
> update even several years old system without re-installation from
> scratch with convenient and half-automated config updates. Of course,
> there are ways for a gradual update of a binary distribution, e.g.
> RHEL-5.0 to RHEL-6.1, but they are too painful to my taste and often
> are not supported by the distribution at all (you''re on your own).
> 
> There is a drawback of course: it takes longer to perform initial
> fine-tuned system setup. As usual, one needs to pay for benefits by
> some overhead in other areas.
> 
> Of course, you mileage may vary depending on your needs. If you are
> happy with your current setup, there are no reasons to change things.
> 
> Best regards,
> Andrew Savchenko
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

Cheers, Andreas
--
Andreas Dilger                       Whamcloud, Inc.
Principal Lustre Engineer            http://www.whamcloud.com/




-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 235 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120605/c524e8d9/attachment.bin

On Thu, 7 Jun 2012 01:55:35 +0400 Andrew Savchenko
wrote:
> Hello Andreas,
> 
> On Wed, 6 Jun 2012 10:01:22 -0600 Andreas Dilger wrote:
> > Note that Lustre 2.2 is not yet designated a maintenance release, so
will not necessarily be getting fixes and updated releases. If this is for a
production system, and you are not prepared to be making your own updates, you
should stick with 2.1.x.
> > 
> > The 2.1.2 release is imminent, but for now 2.1.1 is the latest
maintenance release.
> 
> We''re planning to use Lustre for production purposes in about half
a
> year. Until then it will be used for testing and to create client
> packages and related build system parts.
> 
> So if 2.2 branch will be eventually put to maintenance, half a year
> looks like a good time. If it may never become a maintained branch,
> then I''ll stick with 2.1 branch.
> 
> Best regards,
> Andrew Savchenko

Best regards,
Andrew Savchenko
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120607/88de89f5/attachment.bin

On Thu, 7 Jun 2012 01:55:35 +0400 Andrew Savchenko
wrote:
> On Wed, 6 Jun 2012 10:01:22 -0600 Andreas Dilger wrote:
>> Note that Lustre 2.2 is not yet designated a maintenance release, so
will not necessarily be getting fixes and updated releases. If this is for a
production system, and you are not prepared to be making your own updates, you
should stick with 2.1.x.
>> 
>> The 2.1.2 release is imminent, but for now 2.1.1 is the latest
maintenance release.
> 
> We''re planning to use Lustre for production purposes in about half
a
> year. Until then it will be used for testing and to create client
> packages and related build system parts.
> 
> So if 2.2 branch will be eventually put to maintenance, half a year
> looks like a good time. If it may never become a maintained branch,
> then I''ll stick with 2.1 branch.
Currently there is no plan to move away from 2.1.x as the maintenance branch. 
The next long-term maintenance branch may not be until 2.3 or 2.4, but this will
be decided in the future.  We definitely do not want to discourage testing and
usage of Lustre 2.2 for experienced users that are already expecting to be doing
maintenance themselves, but it has not been getting fixes for known issues that
are landing on master and 2.1.x.

Cheers, Andreas
--
Andreas Dilger                       Whamcloud, Inc.
Principal Lustre Engineer            http://www.whamcloud.com/




-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 235 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20120606/d7cebb87/attachment.bin

FYI: Cray plans to deploy 2.2 at year''s end and therefore
we''ll treat it
as our maintenance branch.  It is true that not much has landed to b2_2 
since 2.2 was released, but Cray will be pushing patches to it including 
some that have already landed to b2_1 for 2.1.x.  It is our hope that we 
can catch it up and b2_2 will be a stable branch for others to use in 
production as Cray will.  We have not yet begun the push process but 
expect to begin shortly.  ORNL, I believe, has similar plans and James 
Simmons has already starting posting patches to b2_2 for review.

Thanks,
-Cory

On 6/6/2012 6:08 PM, Andreas Dilger wrote:
> On Thu, 7 Jun 2012 01:55:35 +0400 Andrew Savchenko wrote:
>> On Wed, 6 Jun 2012 10:01:22 -0600 Andreas Dilger wrote:
>>> Note that Lustre 2.2 is not yet designated a maintenance release,
so will not necessarily be getting fixes and updated releases. If this is for a
production system, and you are not prepared to be making your own updates, you
should stick with 2.1.x.
>>>
>>> The 2.1.2 release is imminent, but for now 2.1.1 is the latest
maintenance release.
>>
>> We''re planning to use Lustre for production purposes in about
half a
>> year. Until then it will be used for testing and to create client
>> packages and related build system parts.
>>
>> So if 2.2 branch will be eventually put to maintenance, half a year
>> looks like a good time. If it may never become a maintained branch,
>> then I''ll stick with 2.1 branch.
>
> Currently there is no plan to move away from 2.1.x as the maintenance
branch.  The next long-term maintenance branch may not be until 2.3 or 2.4, but
this will be decided in the future.  We definitely do not want to discourage
testing and usage of Lustre 2.2 for experienced users that are already expecting
to be doing maintenance themselves, but it has not been getting fixes for known
issues that are landing on master and 2.1.x.
>
> Cheers, Andreas
> --
> Andreas Dilger                       Whamcloud, Inc.
> Principal Lustre Engineer            http://www.whamcloud.com/
>
>
>
>
>
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss