Lustre discuss - Oct 2007 - Lustre 1.6.3 - where are the bug fixes?

OK, I know that there is supposedly some QA before lustre releases and 
that it might be the reason for fixes taking such a long time to 
propagate, but still: It takes too long for fixes to end up in a 
released version...

During our rather limited testing on Ubuntu Dapper (using the Debian 
2.6.18 kernel on servers and pkg-lustre packaging) we''ve run into 
a couple of bugs, most of them with the typical "fix in bugzilla".

The pkg-lustre packaging has six fixes from bugzilla applied, they 
seem to have munged the bug numbers but it seems that only three of 
them are in the 1.6.3 changelog.

We have locally applied fixes from bug 13438 (lustre is totally 
useless without it due to servers OOPS:ing) and 13614. None of them 
seems to be in the 1.6.3 changelog.

So, I''d suggest that CFS gets their act together and starts releasing 
versions more often, if they''d done this during 1.6 development we 
wouldn''t be installing production releases that you can crash after a 
day of testing now.

If QA is the argument for not doing releases more often, consider the 
fact that known broken releases that you have to patch yourself with 
patches hidden in bugzilla isn''t much better.

In reality, I think that doing non-QA''d snapshot releases might be the 
way to go. That is, releases with the useful more-or-less trivial 
fixes that avoids crashes etc. and that will be included in the next 
QA''d release. They would not be suitable for production, but at least 
you can rather easily download the latest snapshot and try on your 
test cluster and see if it fixes the problem(s) you''ve encountered. 
And if it does, we can bug CFS until they get their act together and 
gets a release out with the fix included.

In the end, you have to realise that when you have a production system 
you don''t want to wait for weeks and months for a new release that 
might fix a crash-inducing bug you''re hitting. I say might here, 
because obviously having a fix hidden in bugzilla is no guarantee that 
it''s included in a released version.

In our case we''re not at production yet because of these problems with 
getting fixes out quickly enough. So far we''ve always been able to 
crash lustre 1.6 within days, and that''s after waiting for 1.6 for 
well over a year.

So, I''d like to challenge CFS to get a version of lustre 1.6 (or 1.8, 
whatever) out that proves stable on our small lustre test setup. 
Without patches. In the year of 2007.

Since the "internal QA only" approach obviously isn''t
working, I''d
suggest that you embrace "release early, release often" to get there. 
That means one release per week as long as you have fixes pending to 
get a decent churn on things.


/Nikke
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
  Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se     |    nikke at hpc2n.umu.se
---------------------------------------------------------------------------
  Short cut... the longest distance between two points.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Nikke,
Are you customer of Lustre Support? I don''t have you listed as a
supported
customer.  Maybe we should arrange a discussion about how we could assist
you more effectively.

Best regards,
Kevin 
-- 
P. Kevin Canady
Director, Business Development
Lustre Group (Formerly CFS)
Sun Microsystems, Inc.
O: 415.928.3633
C: 415.505.7701


On 10/11/07 11:53 PM, "Niklas Edmundsson" <Niklas.Edmundsson at
hpc2n.umu.se>
wrote:
> 
> OK, I know that there is supposedly some QA before lustre releases and
> that it might be the reason for fixes taking such a long time to
> propagate, but still: It takes too long for fixes to end up in a
> released version...
> 
> During our rather limited testing on Ubuntu Dapper (using the Debian
> 2.6.18 kernel on servers and pkg-lustre packaging) we''ve run into
> a couple of bugs, most of them with the typical "fix in
bugzilla".
> 
> The pkg-lustre packaging has six fixes from bugzilla applied, they
> seem to have munged the bug numbers but it seems that only three of
> them are in the 1.6.3 changelog.
> 
> We have locally applied fixes from bug 13438 (lustre is totally
> useless without it due to servers OOPS:ing) and 13614. None of them
> seems to be in the 1.6.3 changelog.
> 
> So, I''d suggest that CFS gets their act together and starts
releasing
> versions more often, if they''d done this during 1.6 development we
> wouldn''t be installing production releases that you can crash
after a
> day of testing now.
> 
> If QA is the argument for not doing releases more often, consider the
> fact that known broken releases that you have to patch yourself with
> patches hidden in bugzilla isn''t much better.
> 
> In reality, I think that doing non-QA''d snapshot releases might be
the
> way to go. That is, releases with the useful more-or-less trivial
> fixes that avoids crashes etc. and that will be included in the next
> QA''d release. They would not be suitable for production, but at
least
> you can rather easily download the latest snapshot and try on your
> test cluster and see if it fixes the problem(s) you''ve
encountered.
> And if it does, we can bug CFS until they get their act together and
> gets a release out with the fix included.
> 
> In the end, you have to realise that when you have a production system
> you don''t want to wait for weeks and months for a new release that
> might fix a crash-inducing bug you''re hitting. I say might here,
> because obviously having a fix hidden in bugzilla is no guarantee that
> it''s included in a released version.
> 
> In our case we''re not at production yet because of these problems
with
> getting fixes out quickly enough. So far we''ve always been able to
> crash lustre 1.6 within days, and that''s after waiting for 1.6 for
> well over a year.
> 
> So, I''d like to challenge CFS to get a version of lustre 1.6 (or
1.8,
> whatever) out that proves stable on our small lustre test setup.
> Without patches. In the year of 2007.
> 
> Since the "internal QA only" approach obviously isn''t
working, I''d
> suggest that you embrace "release early, release often" to get
there.
> That means one release per week as long as you have fixes pending to
> get a decent churn on things.
> 
> 
> /Nikke

Hmmm.   This is the approach that has always frightened us away from  
Lustre.   Is there one version of Lustre for paying support customers  
and another for those who, for whatever reason, can''t or don''t
want
to pay for support.   What do you mean when you say "assist you more  
effectively"?   Does that mean you will apply the patches in the  
bugzilla database for him?   How does that help if he can do it  
himself?   The point Nikke is making is that a release labeled  
"production" should run for more than a few hours to a day without  
crashing.   And, if you have a "production" release that crashes that
readily AND there are known fixes, it seems like they should be  
readily available in a format that is easy to apply (as opposed to  
digging through bug reports and applying code fixes by hand).

For a lot of very good reasons, we would like to go to lustre.    
There is much to like about it.  However, we run OFED 1.2 on our  
cluster and need Lustre 1.6.2+ for OFED support.   So far, our  
attempts to test this version of lustre on our 400+ node IB cluster  
have resulted in impressive performance and scalability and...lots of  
crashes (mballoc) and a corrupt file system that neither e2fsck nor  
lfsck could fix.   It is too bad because it seems that lustre is just  
a few fixes away from having one of the most amazing open source  
packages in the Linux universe.

We wish you the best and hope that we will be able to use Lustre in  
the near future.

Charlie Taylor
UF HPC Center


On Oct 12, 2007, at 3:20 AM, Kevin Canady wrote:
> Nikke,
> Are you customer of Lustre Support? I don''t have you listed as a  
> supported
> customer.  Maybe we should arrange a discussion about how we could  
> assist
> you more effectively.
>
> Best regards,
> Kevin
> -- 
> P. Kevin Canady
> Director, Business Development
> Lustre Group (Formerly CFS)
> Sun Microsystems, Inc.
> O: 415.928.3633
> C: 415.505.7701
>
>
> On 10/11/07 11:53 PM, "Niklas Edmundsson"  
> <Niklas.Edmundsson at hpc2n.umu.se>
> wrote:
>
>>
>> OK, I know that there is supposedly some QA before lustre releases  
>> and
>> that it might be the reason for fixes taking such a long time to
>> propagate, but still: It takes too long for fixes to end up in a
>> released version...
>>
>> During our rather limited testing on Ubuntu Dapper (using the Debian
>> 2.6.18 kernel on servers and pkg-lustre packaging) we''ve run
into
>> a couple of bugs, most of them with the typical "fix in
bugzilla".
>>
>> The pkg-lustre packaging has six fixes from bugzilla applied, they
>> seem to have munged the bug numbers but it seems that only three of
>> them are in the 1.6.3 changelog.
>>
>> We have locally applied fixes from bug 13438 (lustre is totally
>> useless without it due to servers OOPS:ing) and 13614. None of them
>> seems to be in the 1.6.3 changelog.
>>
>> So, I''d suggest that CFS gets their act together and starts
releasing
>> versions more often, if they''d done this during 1.6
development we
>> wouldn''t be installing production releases that you can crash
after a
>> day of testing now.
>>
>> If QA is the argument for not doing releases more often, consider the
>> fact that known broken releases that you have to patch yourself with
>> patches hidden in bugzilla isn''t much better.
>>
>> In reality, I think that doing non-QA''d snapshot releases
might be
>> the
>> way to go. That is, releases with the useful more-or-less trivial
>> fixes that avoids crashes etc. and that will be included in the next
>> QA''d release. They would not be suitable for production, but
at least
>> you can rather easily download the latest snapshot and try on your
>> test cluster and see if it fixes the problem(s) you''ve
encountered.
>> And if it does, we can bug CFS until they get their act together and
>> gets a release out with the fix included.
>>
>> In the end, you have to realise that when you have a production  
>> system
>> you don''t want to wait for weeks and months for a new release
that
>> might fix a crash-inducing bug you''re hitting. I say might
here,
>> because obviously having a fix hidden in bugzilla is no guarantee  
>> that
>> it''s included in a released version.
>>
>> In our case we''re not at production yet because of these
problems
>> with
>> getting fixes out quickly enough. So far we''ve always been
able to
>> crash lustre 1.6 within days, and that''s after waiting for 1.6
for
>> well over a year.
>>
>> So, I''d like to challenge CFS to get a version of lustre 1.6
(or 1.8,
>> whatever) out that proves stable on our small lustre test setup.
>> Without patches. In the year of 2007.
>>
>> Since the "internal QA only" approach obviously
isn''t working, I''d
>> suggest that you embrace "release early, release often" to
get there.
>> That means one release per week as long as you have fixes pending to
>> get a decent churn on things.
>>
>>
>> /Nikke
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss

On Fri, Oct 12, 2007 at 05:49:44AM -0400, Charles Taylor
wrote:
> 
> Hmmm.   This is the approach that has always frightened us away from  
> Lustre.   Is there one version of Lustre for paying support customers  
> and another for those who, for whatever reason, can''t or
don''t want
> to pay for support.
That''s my personnal opinion too: it''s fine if clusterfs.com
charges for support
but it''s pretty frigthening if the OSS version if different from the
paying version.

Cheers,

Tru
-- 
Dr Tru Huynh          | http://www.pasteur.fr/recherche/unites/Binfs/
mailto:tru at pasteur.fr | tel/fax +33 1 45 68 87 37/19
Institut Pasteur, 25-28 rue du Docteur Roux, 75724 Paris CEDEX 15 France

On Fri, 12 Oct 2007, Kevin Canady wrote:
> Nikke,
> Are you customer of Lustre Support? I don''t have you listed as a
supported
> customer.  Maybe we should arrange a discussion about how we could assist
> you more effectively.
Not anymore. Back when we had support we didn''t get much useful 
answers to our support cases, they entered into bugzilla and if we 
were lucky we got some kind of response. And given that we didn''t have 
lustre in production (partly due to hardware issues, so that''s not 
entirely due to lustre) we didn''t see any reason to continue paying 
for support.

If we get to the point that lustre 1.6 (or 1.8, or...) seems to be 
production ready we might use it. If we use it we might purchase 
support again.

But, support or not this doesn''t answer the fundamental issue:
It''s a
long time between lustre releases, a fact that hasn''t changed since 
the public CVS access was revoked.

/Nikke
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
  Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se     |    nikke at hpc2n.umu.se
---------------------------------------------------------------------------
  "He will not be permanently damaged." - Vader
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

On Oct 12, 2007  08:53 +0200, Niklas Edmundsson wrote:
> OK, I know that there is supposedly some QA before lustre releases and 
> that it might be the reason for fixes taking such a long time to 
> propagate, but still: It takes too long for fixes to end up in a 
> released version...
I tend to agree that it does take a long time between releases, and
we are definitely trying to improve this.
> The pkg-lustre packaging has six fixes from bugzilla applied, they 
> seem to have munged the bug numbers but it seems that only three of 
> them are in the 1.6.3 changelog.
Could you please indicate which bugs are missing?
> We have locally applied fixes from bug 13438 (lustre is totally 
> useless without it due to servers OOPS:ing) and 13614. None of them 
> seems to be in the 1.6.3 changelog.
The 13614 patch is actually in 1.6.3 under the original bug number
where the patch came from (bug 13596).

As for 13438, I cannot explain why this didn''t make it into the 1.6.3
release, since it was definitely ready in time.


Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

On Oct 12, 2007  12:11 +0200, Tru Huynh wrote:
> On Fri, Oct 12, 2007 at 05:49:44AM -0400, Charles Taylor wrote:
> > Hmmm.   This is the approach that has always frightened us away from  
> > Lustre.   Is there one version of Lustre for paying support customers
> > and another for those who, for whatever reason, can''t or
don''t want
> > to pay for support.
> 
> That''s my personnal opinion too: it''s fine if
clusterfs.com charges for
> support but it''s pretty frigthening if the OSS version if
different from
> the paying version. 
I''m not sure where you got this opinion, but it is incorrect.  There is
a single version of Lustre that we release.  Why would we ever want to
have the overhead of maintaining two separate releases?

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

On Oct 12, 2007  05:49 -0400, Charles Taylor wrote:
> Hmmm.   This is the approach that has always frightened us away from  
> Lustre.   Is there one version of Lustre for paying support customers  
> and another for those who, for whatever reason, can''t or
don''t want
> to pay for support.
No, there is a single version that we release for everyone.
> What do you mean when you say "assist you more  
> effectively"?   Does that mean you will apply the patches in the  
> bugzilla database for him?   How does that help if he can do it  
> himself?
The issue is that we are a company that is paying a lot of people to
maintain and develop Lustre, and without paying customers none of this
would get very far.  It''s great that we can give Lustre away freely for
anyone, but the truth is that customers who are paying for support have
priority in terms of getting attention to their bugs.

In terms of how it can help Nikke to have support with CFS - it means
we can afford to have someone smart like Alex on our staff to fix this
problem in the first place.  The fact that the fix in question didn''t
make
it into 1.6.3 is an oversight on our part, as the patch was definitely
ready in time for the release.  We will of course include it in the next
release, possibly making a point release with this fix sooner.
> For a lot of very good reasons, we would like to go to lustre.    
> There is much to like about it.  However, we run OFED 1.2 on our  
> cluster and need Lustre 1.6.2+ for OFED support.   So far, our  
> attempts to test this version of lustre on our 400+ node IB cluster  
> have resulted in impressive performance and scalability and...lots of  
> crashes (mballoc) and a corrupt file system that neither e2fsck nor  
> lfsck could fix.   It is too bad because it seems that lustre is just  
> a few fixes away from having one of the most amazing open source  
> packages in the Linux universe.
I''m sorry to hear you are having problems, possibly the same ones being
discussed here.  As for OFED 1.2, this will also be available in the
1.4.12 release, if that is of interest to you, and of course we are
working to address issues with 1.6 as well.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Am Freitag, 12. Oktober 2007 12:29:08 schrieb Andreas
Dilger:
> On Oct 12, 2007  08:53 +0200, Niklas Edmundsson wrote:
> > OK, I know that there is supposedly some QA before lustre releases and
> > that it might be the reason for fixes taking such a long time to
> > propagate, but still: It takes too long for fixes to end up in a
> > released version...
>
> I tend to agree that it does take a long time between releases, and
> we are definitely trying to improve this.
>
> > The pkg-lustre packaging has six fixes from bugzilla applied, they
> > seem to have munged the bug numbers but it seems that only three of
> > them are in the 1.6.3 changelog.
>
> Could you please indicate which bugs are missing?
Hello, :)

I''m one of the debian maintainers and I started today to package 1.6.3
and
look through your bugzilla in order to get all patches which are necessary 
to build a stable package.

While doing so i detect that your changelog on your website is not 
completly correct. (It''s missing some bugs which are closed in the
1.6.3
release):
This bugs are:
 - 13610
 - 12475
 - 5491
 - 11880

I guess someone has forgotten to mention them there. I would suggest to 
update this information.

Greetings
Patrick Winnertz

-- 
Patrick Winnertz
Tel.: +49 (0) 2161 / 4643 - 0

credativ GmbH, HRB M?nchengladbach 12080
Hohenzollernstr. 133, 41061 M?nchengladbach
Gesch?ftsf?hrung: Dr. Michael Meskes, J?rg Folz

On Fri, 12 Oct 2007, Niklas Edmundsson wrote:
> In reality, I think that doing non-QA''d snapshot releases might be
the
> way to go. That is, releases with the useful more-or-less trivial
> fixes that avoids crashes etc. and that will be included in the next
> QA''d release.
Yes, we find ourselves carrying a lot of bugfix patches in our
internal Lustre releases (50+ in our last production release based 
on 1.4.8, already 30+ in our release planned for january based on 1.6.2).

I think it would be useful if CFS would freeze features in a release at
some point and do bugfix-only releases.  For example, we''re trying to
get 1.6.2 stablized for production use and we know that many bugfixes 
that we need are in 1.6.3, but we have to backport them because we
don''t
want to destablize our effort by taking the features in 1.6.3.

I will say in CFS''s defense that we abuse Lustre in ways that not all
customers do, and so in some cases a bugfix to us might be controversial
to apply to a stable release series.  Not true of all the patches
wer are carrying though.

Jim

On Oct 12, 2007  18:20 +0200, Patrick Winnertz wrote:
> Am Freitag, 12. Oktober 2007 12:29:08 schrieb Andreas Dilger:
> > On Oct 12, 2007  08:53 +0200, Niklas Edmundsson wrote:
> > > OK, I know that there is supposedly some QA before lustre
releases and
> > > that it might be the reason for fixes taking such a long time to
> > > propagate, but still: It takes too long for fixes to end up in a
> > > released version...
> >
> > I tend to agree that it does take a long time between releases, and
> > we are definitely trying to improve this.
> >
> > > The pkg-lustre packaging has six fixes from bugzilla applied,
they
> > > seem to have munged the bug numbers but it seems that only three
of
> > > them are in the 1.6.3 changelog.
> >
> > Could you please indicate which bugs are missing?
> 
> I''m one of the debian maintainers and I started today to package
1.6.3 and
> look through your bugzilla in order to get all patches which are necessary 
> to build a stable package.
Actually, what I''M interested in is which patches you have in the
Debian
packages that are not in the CFS release.  We obviously want to include
such fixes into our release, and I agree that our patch-tracking process
can miss fixes on occasion, so having feedback from the Debian maintainers
would definitely help avoid such issues.
> While doing so i detect that your changelog on your website is not 
> completly correct. (It''s missing some bugs which are closed in the
1.6.3
> release):
> This bugs are:
>  - 13610
This relates to a fix made to a patch that was itself landed before 1.6.3
was released, so our policy is to not mention this as a separate fix in
the ChangeLog, to avoid clutter.
>  - 12475
This one is only an error message fix.  We don''t necessarily add every
change into the ChangeLog, as this makes it too noisy to see the important
fixes going in.
>  - 5491
>  - 11880
These two were landed 1.6.3, but the proper processes weren''t being
followed.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Hi Andreas,

First, I''d like to say that I tend to agree with Niklas''
remarks in
general. I think that having stable, QA''d releases is important, but 
considering that most of the Lustre audience is made of highly 
technical people running large installations and needing to run their 
own tests, having more frequent (not fully QA''d) releases would be a 
great deal of help.

On Friday 12 October 2007 03:29:08 am Andreas Dilger
wrote:
> Could you please indicate which bugs are missing?
We''re also missing 13438, which is really a stopper for this release.
> The 13614 patch is actually in 1.6.3 under the original bug number
> where the patch came from (bug 13596).
Good to know. It wasn''t clear since 13596 is not public, and the 
discussion in 13614 didn''t indicate any progress on 13596 resolution, 
neither gave any confirmation that both bugs were resolved by the same 
fix.
> As for 13438, I cannot explain why this didn''t make it into the
1.6.3
> release, since it was definitely ready in time.
What about a 1.6.3.1 including 13438? ;)

Cheers,
-- 
Kilian

Hi Jim,

On Friday 12 October 2007 09:31:46 am Jim Garlick wrote:
> I will say in CFS''s defense that we abuse Lustre in ways that not
all
> customers do, 
We all abuse Lustre one way or another. :)

Considering the astonishing performance Lustre gives our users, and 
since most of them are not accustomed to that from their past 
experience, they sometimes tend to see it as a ressource they can 
stress shamelessly. Which at least gives us the opportunity to fill up 
some bug reports. :)

Cheers,
-- 
Kilian

On Fri, 12 Oct 2007, Kilian CAVALOTTI wrote:
>> The 13614 patch is actually in 1.6.3 under the original bug number
>> where the patch came from (bug 13596).
>
> Good to know. It wasn''t clear since 13596 is not public, and the
> discussion in 13614 didn''t indicate any progress on 13596
resolution,
> neither gave any confirmation that both bugs were resolved by the same
> fix.
I agree. This is why I prefer that all bug numbers are mentioned in 
the changelog (be it a separate long-format changelog).

Actually, I''d prefer if the bugs themselves were public in the 
bugzilla once identified. The reason for having bugs private should be 
that the discussion/testcase/etc needs to be private, not that the 
actual bug itself is a secret, right?
>> As for 13438, I cannot explain why this didn''t make it into
the 1.6.3
>> release, since it was definitely ready in time.
>
> What about a 1.6.3.1 including 13438? ;)
What about a more general policy that pure bugfixes yields a new 
point-release? As Jim Garlick points out there are usually new 
features interspersed with the bugfixes, which has the potential of 
destabilizing the whole thing.

I''d prefer if point releases were produced until the thing was 
relatively stable (meaning that people can actually run it without 
applying their own batch of fixes), and then new features were 
included. Right now the only way of stabilizing lustre seems to be for 
each site to run into bugs, and dig in maillists/bugzilla to find the 
fixes.

So, get 1.6.3.1 out the door ASAP, and keep producing point-releases 
weekly until you''ve closed all "my server/client goes OOPS"
bugs. I
can almost guarantee that if CFS shows that it intends to fix these 
bugs swiftly people will deploy new point releases and report the bugs 
they find.


/Nikke
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
  Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se     |    nikke at hpc2n.umu.se
---------------------------------------------------------------------------
  *   <- Tribble     *\ <- Tribble Going Skiing
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

On Fri, Oct 12, 2007 at 09:31:46AM -0700, Jim Garlick
wrote:
> On Fri, 12 Oct 2007, Niklas Edmundsson wrote:
> 
> > In reality, I think that doing non-QA''d snapshot releases
might be the
> > way to go. That is, releases with the useful more-or-less trivial
> > fixes that avoids crashes etc. and that will be included in the next
> > QA''d release.
> 
> Yes, we find ourselves carrying a lot of bugfix patches in our
> internal Lustre releases (50+ in our last production release based 
> on 1.4.8, already 30+ in our release planned for january based on 1.6.2).
> 
> I think it would be useful if CFS would freeze features in a release at
> some point and do bugfix-only releases.  For example, we''re trying
to
> get 1.6.2 stablized for production use and we know that many bugfixes 
> that we need are in 1.6.3, but we have to backport them because we
don''t
> want to destablize our effort by taking the features in 1.6.3.
This sounds like a lot of duplicated effort that if CFS did not want to
support could be managed by a decent distributed source control system
(either git or mercurial), and the interested ''power users''.

It''s also annoying that a quick web search for ''Lustre
CVS'' turns up a
bunch of ancient or dead links. If I could get the latest Lustre
development version from a *public* source control server, I''d be using
Lustre for testing remote filesystem access over InfiniBand WAN links.
As it is, the only parallel filesystem I can easily find with a public
source control server is PVFS, so that''s what I''m using, and
it works
quite nicely.