Lustre devel - Apr 2010 - Build system and portability

(I wanted to start some discussions on portability.  This is one of the
earliest things that I ran into for the MacOS X port).

So, the build system was one of my earliest challenges, and I wanted to get
some ideas from other people.

Aside from the various Linux-specific autoconf tests (those are relatively
straightforward to deal with), one big issue that has cropped up is the
way that the build system ends up building Makefiles.

Specifically, there seems to be a disconnect between Makefile.in and
autoMakefile.am.  Let''s take lustre/obdclass as an example.

The object files for the kernel module for Linux are listed in Makefile.in
(they''re in the variables obdclass-linux-objs and obdclass-all-objs,
and
those are combined into obdclass-objs).  However, the actual definitions
of the modules to build get set in autoMakefile.am (by setting the
modulefs_DATA variable).

For MacOS X, the kext is listed under macos_PROGRAM, and the _sources_
are listed (instead of objects).  This all happens in autoMakefile.am
(and is ifdef''d protected for DARWIN only); it doesn''t use
anything in
Makefile.in.  The special flags needed for MacOS X module builds are
overridden for those builds by setting obdclass_CFLAGS (or whatever is
the appropriate name for the module you''re building).

This feels ... inelegant to me.  For one, if new source code files are added,
you have to add them two places (this bit me a number of times).  And
it''s
a lot of extra work to support different OS''s.

Now I inherited this build system and it was 90% of the way there, so I
can''t complain about it that much.  But I think we could do better.

What do people think about what this current mess?  Should it be normalized?
What''s the right way to clean this up?  If it''s to support
module building
similar to the way Linux kernel modules are declared in the
Makefile/autoMakefile, then I''m okay with that.

--Ken

I''m the first one to admit that our build system is crufty. Brian  
Murrell is probably going to be the second, and he is the lucky one  
that knows the most about it.

The legacy is that it had to be complex in order to allow building  
against both 2.4 and 2.6 kernels, but that need is long gone but we  
still have the complexity left behind.

I would be thrilled to have someone interested/knowledgable enough to  
update it to be more normal, since you want it to be portable to Macos  
and we want it to be portable to solaris and windows.

Is this something you are interested to invest some time on fixing, or  
is it more a question of curiosity as to how we got into this Rube  
Goldberg build system?

Cheers, Andreas

On 2010-04-23, at 10:27, Ken Hornstein <kenh at cmf.nrl.navy.mil> wrote:
> (I wanted to start some discussions on portability.  This is one of  
> the
> earliest things that I ran into for the MacOS X port).
>
> So, the build system was one of my earliest challenges, and I wanted  
> to get
> some ideas from other people.
>
> Aside from the various Linux-specific autoconf tests (those are  
> relatively
> straightforward to deal with), one big issue that has cropped up is  
> the
> way that the build system ends up building Makefiles.
>
> Specifically, there seems to be a disconnect between Makefile.in and
> autoMakefile.am.  Let''s take lustre/obdclass as an example.
>
> The object files for the kernel module for Linux are listed in  
> Makefile.in
> (they''re in the variables obdclass-linux-objs and
obdclass-all-objs,
> and
> those are combined into obdclass-objs).  However, the actual  
> definitions
> of the modules to build get set in autoMakefile.am (by setting the
> modulefs_DATA variable).
>
> For MacOS X, the kext is listed under macos_PROGRAM, and the _sources_
> are listed (instead of objects).  This all happens in autoMakefile.am
> (and is ifdef''d protected for DARWIN only); it doesn''t
use anything in
> Makefile.in.  The special flags needed for MacOS X module builds are
> overridden for those builds by setting obdclass_CFLAGS (or whatever is
> the appropriate name for the module you''re building).
>
> This feels ... inelegant to me.  For one, if new source code files  
> are added,
> you have to add them two places (this bit me a number of times).   
> And it''s
> a lot of extra work to support different OS''s.
>
> Now I inherited this build system and it was 90% of the way there,  
> so I
> can''t complain about it that much.  But I think we could do
better.
>
> What do people think about what this current mess?  Should it be  
> normalized?
> What''s the right way to clean this up? If it''s to support
module
> building
> similar to the way Linux kernel modules are declared in the
> Makefile/autoMakefile, then I''m okay with that.
>
> --Ken
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel

>Is this something you are interested to invest some time on fixing, or  
>is it more a question of curiosity as to how we got into this Rube  
>Goldberg build system?
I am definitely interested in helping to clean up the current build
system.  Obviously history is helpful to understand how things ended
up where they are now, and I''m not that experienced on how to
configure a build system to create Linux kernel modules so the goal of
my note was to just start the ball rolling.

I think it''s obvious that the autoconf system needs some work as well;
no one disagrees, do they?

I guess the first thing that comes to mind is ... what''s the deal with
Makefile.in and autoMakefile.am?  Why not just one Makefile.am?

--Ken

On Sun, 2010-04-25 at 19:32 -0400, Ken Hornstein wrote: 
> 
> I think it''s obvious that the autoconf system needs some work as
well;
> no one disagrees, do they?
The autoconf portion is not terribly bad, I think.  It could probably
use a bit of macro reorganization, and some more reasonable defaults for
things like kernel, etc. but I am working on the latter currently.  It
seemed a real mess to me at first, but the more I worked with it, the
more I understood the organization of it.
> I guess the first thing that comes to mind is ... what''s the deal
with
> Makefile.in and autoMakefile.am?  Why not just one Makefile.am?
I have not really delved terribly deeply into the organization of an
external (linux) kernel module source tree, but I am at least familiar
with it.

My (self-attained, admittedly, so take it with a grain of salt)
understanding of the reason for both Makefiles and autoMakefiles is that
the Makefiles are what the kernel build system uses to build the kernel
modules portion of lustre and the autoMakefiles are for everything else.

Why we cannot/didn''t just put the modules into it''s own tree
with kernel
build system Makefiles and keep everything in a separate tree with their
own non-kernel build system Makefiles... I am not sure, but as Andreas
mentioned, is likely just due to the history and the baggage that has
not been shed when support for historical systems was dropped.

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-devel/attachments/20100426/82b83503/attachment.bin

>The autoconf portion is not terribly bad, I think.  It could probably
>use a bit of macro reorganization, and some more reasonable defaults for
>things like kernel, etc. but I am working on the latter currently.  It
>seemed a real mess to me at first, but the more I worked with it, the
>more I understood the organization of it.
Forgive me for implying otherwise; it''s really more a bunch of
portability cleanup issues.  Like not doing Linux kernel feature tests
on non-Linux systems, that sort of thing (I guess those don''t
technically
harm anything, but there were a few things that would fail and those needed
to be system-dependent).
>Why we cannot/didn''t just put the modules into it''s own
tree with kernel
>build system Makefiles and keep everything in a separate tree with their
>own non-kernel build system Makefiles... I am not sure, but as Andreas
>mentioned, is likely just due to the history and the baggage that has
>not been shed when support for historical systems was dropped.
Again, I''m not the expert on how kernel module building works ... but
there seems to be some interdependency in the current Lustre setup (the
Makefiles include Rules, which includes autoMakefile).  I would just be
happy if we could get down to one Makefile; is that possible, given the
way that Linux kernel module builds work?

Oh, as long as we''re on the subject ... when I file portability bugs
using bugzilla, I am never happy with the component I have to pick to
file the bug under (I end up with something like "integration").  Can
we get a Bugzilla component titled "portability"?  Or if people are
happy with "integration" (or would rather I use something else), I
guess that''s okay as well.

--Ken

On Mon, 2010-04-26 at 14:15 -0400, Ken Hornstein wrote: 
> 
> Forgive me for implying otherwise;
No worries.  I totally understand where you are at with all of that.
> it''s really more a bunch of
> portability cleanup issues.
Indeed.  More or less what I was saying.
> Like not doing Linux kernel feature tests
> on non-Linux systems,
Right.  The closer you look, you will notice there is an intention to
group macros that are O/S specific into their own files and macro
calling groups, but I would not at all be surprised to learn that there
is some (or even a lot of) leakage, as the reality is that we don''t
really build on anything but Linux at this time and therefore we don''t
really have an opportunity to catch that leakage.

I suspect as we get deeper into Solaris, a lot of this will get weeded
out.  Or as you contribute your patches, equally so.
> Again, I''m not the expert on how kernel module building works ...
Me neither.  :-)  I have a basic understanding, but that''s it.
> but
> there seems to be some interdependency in the current Lustre setup (the
> Makefiles include Rules, which includes autoMakefile).
Yes, indeed.
> I would just be
> happy if we could get down to one Makefile; is that possible, given the
> way that Linux kernel module builds work?
Hrm.  Just given the existing relationship between Makefiles, the Rules
file and autoMakefiles, it seems probably not, without a lot of
duplication (and the ongoing maintenance nightmare) throughout
Makefiles.  I am happy to be proven wrong though.

I think the problem is that ultimately the code that is in the tree
needs to build for multiple platforms, including both traditional (Linux
kernel) lustre and liblustre.  So it''s not just a case of separate bits
of code exclusive to the linux kernel and other separate bits exclusive
to userspace, but that a large part of the code that can be built into
linux modules (with it''s particular Makefile requirements) is also used
to build liblustre.

I would think that MacOS/X has it''s own kernel module build sequences
such that the Makefiles that are engineered for the Linux kernel build
tree are not suitable.  Or are the Linux module Makefiles also working
for the MacOS/X kernel?  That would add a third variant, I guess.  I
know zilch about building MacOS/X modules, so I can''t really judge.
> Oh, as long as we''re on the subject ... when I file portability
bugs
> using bugzilla, I am never happy with the component I have to pick to
> file the bug under (I end up with something like "integration"). 
Can
> we get a Bugzilla component titled "portability"?  Or if people
are
> happy with "integration" (or would rather I use something else),
I
> guess that''s okay as well.
Well, do as well as you can with the component, but indeed, there are a
lot of cases that are simply not a clear case of choosing one of those
components so choose something "close".  For this sort of Makefile
stuff
you are doing, I think there is a component for "build system" or some
such.  That would likely be appropriate.

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-devel/attachments/20100426/c877fb18/attachment.bin

On 2010-04-26, at 15:08, Brian J. Murrell wrote:
> On Mon, 2010-04-26 at 14:15 -0400, Ken Hornstein wrote: 
>> it''s really more a bunch of
>> portability cleanup issues.
> 
> Indeed.  More or less what I was saying.
> 
>> Like not doing Linux kernel feature tests on non-Linux systems,
> 
> Right.  The closer you look, you will notice there is an intention to
> group macros that are O/S specific into their own files and macro
> calling groups, but I would not at all be surprised to learn that there
> is some (or even a lot of) leakage, as the reality is that we
don''t
> really build on anything but Linux at this time and therefore we
don''t
> really have an opportunity to catch that leakage.
Exactly what I was going to say.  I don''t think there is any reason we
would want to do it in a bad way, just that people weren''t thinking of
cross-platform portability when they added some check, and didn''t put
it in the right file at the time.
> I suspect as we get deeper into Solaris, a lot of this will get weeded
> out.  Or as you contribute your patches, equally so.
I''m happy to see them as soon as they are ready (which seems to be on
the client for MacOS, and is being done on the server for Solaris).  No point in
duplicating effort.
>> I would just be
>> happy if we could get down to one Makefile; is that possible, given the
>> way that Linux kernel module builds work?
> 
> Hrm.  Just given the existing relationship between Makefiles, the Rules
> file and autoMakefiles, it seems probably not, without a lot of
> duplication (and the ongoing maintenance nightmare) throughout
> Makefiles.  I am happy to be proven wrong though.
> 
> I think the problem is that ultimately the code that is in the tree
> needs to build for multiple platforms, including both traditional (Linux
> kernel) lustre and liblustre.  So it''s not just a case of separate
bits
> of code exclusive to the linux kernel and other separate bits exclusive
> to userspace, but that a large part of the code that can be built into
> linux modules (with it''s particular Makefile requirements) is also
used
> to build liblustre.
Actually, if we could get liblustre to become "just another client
arch" (like MacOS and WinNT) I think it would do wonders for the
portability as well.  As it stands today, liblustre is kind of a mismash of
Linux emulation and its own thing.
> I would think that MacOS/X has it''s own kernel module build
sequences
> such that the Makefiles that are engineered for the Linux kernel build
> tree are not suitable.  Or are the Linux module Makefiles also working
> for the MacOS/X kernel?  That would add a third variant, I guess.  I
> know zilch about building MacOS/X modules, so I can''t really
judge.
I suspect that we might be able to get something like a "Makefile
fragment" that is containing all of the build targets, that is included in
an arch-specific top-level Makefile to specify exactly how those files will be
built.  Sadly, I don''t know anything about this area.
>> Oh, as long as we''re on the subject ... when I file
portability bugs
>> using bugzilla, I am never happy with the component I have to pick to
>> file the bug under (I end up with something like
"integration").  Can
>> we get a Bugzilla component titled "portability"?  Or if
people are
>> happy with "integration" (or would rather I use something
else), I
>> guess that''s okay as well.
> 
> Well, do as well as you can with the component, but indeed, there are a
> lot of cases that are simply not a clear case of choosing one of those
> components so choose something "close".  For this sort of
Makefile stuff
> you are doing, I think there is a component for "build system" or
some
> such.  That would likely be appropriate.

The intention of "component" is e.g. "what part of the code the
defect is in (e.g. client, server, quota, etc).  I''d think most of your
problems are in "Build/Release Engineering" and "OS X",
though some will be "llite/mdc", "llite/vfs", and
"IO/llite/lov/osc".

In this light, it would be interesting to know whether you were able to re-use
parts of the lustre/llite code (e.g. llite_lib.c) and it should be moved into a
shared directory like lustre/lclient (which is supposed to be entirely
portable).  Spending some time to incrementally move (and get upstream) parts of
the llite code that are portable between linux and macos would ensure that the
ongoing maintenance is reduced.  The problem with making too many changes in a
port and not returning them upstream is that eventually the code diverges so
much that the ongoing

Cheers, Andreas
--
Andreas Dilger
Lustre Technical Lead
Oracle Corporation Canada Inc.

>> I would just be
>> happy if we could get down to one Makefile; is that possible, given the
>> way that Linux kernel module builds work?
>
>Hrm.  Just given the existing relationship between Makefiles, the Rules
>file and autoMakefiles, it seems probably not, without a lot of
>duplication (and the ongoing maintenance nightmare) throughout
>Makefiles.  I am happy to be proven wrong though.
You mean, as opposed to the duplication and maintenance nightmare
that already exists? :-)

I guess my largest issue is that the files necessary to build a
kernel module have to be listed twice (one for the Linux kernel
module, once for the MacOS X kernel modules); that caused a ton of
issues that had to be tracked down and it was rather tedious.
If we could get rid of that, then I''d mostly be happy.
>I would think that MacOS/X has it''s own kernel module build
sequences
>such that the Makefiles that are engineered for the Linux kernel build
>tree are not suitable.  Or are the Linux module Makefiles also working
>for the MacOS/X kernel?  That would add a third variant, I guess.  I
>know zilch about building MacOS/X modules, so I can''t really judge.
Heh, actually, the code mostly existed and was close to working
(well, other than plenty of files had been added in 5 years).  You
can see it in the autoMakefiles; look for ifdef DARWIN and you''ll
see what I mean.

Kernel modules on MacOS X (assuming you''re not using Xcode environment,
which I''m not) are built using the normal compilers and your own
Makefiles;
they just have different flags for compilation and linking.  The MacOS X
kernel modules are built using relatively standard automake descriptions
(see macos_PROGRAM).

--Ken

>I suspect that we might be able to get something like a "Makefile
fragment"
>that is containing all of the build targets, that is included in an
>arch-specific top-level Makefile to specify exactly how those files will
>be built.  Sadly, I don''t know anything about this area.
I am thinking along these lines as well; the wrinkle is the way that
the Linux module building process works.
>The intention of "component" is e.g. "what part of the code
the defect is in
>(e.g. client, server, quota, etc).  I''d think most of your problems
are in
>"Build/Release Engineering" and "OS X", though some will
be "llite/mdc",
>"llite/vfs", and "IO/llite/lov/osc".
Well, take a look at bug 22692 as an example.  That doesn''t feel like
build/release engineering.  And it''s not really MacOS X either.

Another example: if I include a bunch of patches to clean up lock leaks
in libcfs ... where should that go?  Perhaps it''s not a huge deal; I
can
just take my best guess at it and go from there.

(Actually, can we add MacOS X as one of the "OS" choices?).
>In this light, it would be interesting to know whether you were able to
>re-use parts of the lustre/llite code (e.g. llite_lib.c) and it should be
>moved into a shared directory like lustre/lclient (which is supposed to be
>entirely portable).  Spending some time to incrementally move (and get
>upstream) parts of the llite code that are portable between linux and
>macos would ensure that the ongoing maintenance is reduced.  The problem
>with making too many changes in a port and not returning them upstream
>is that eventually the code diverges so much that the ongoing 
Sadly, I could not use anything from llite_lib.c (or anything from
llite, really) at all; it''s way Linux-specific.  Given the way the
code is structured, I''m not sure there is any alternative; Linux
is so different in terms of filesystem implementation I don''t see
very much in llite that could be made generic (I guess I could
imagine some generic stuff, but it would probably be a mess of
ifdefs).  It''s actually even worse than that; the "generic"
code
in obdclass/mount.c is also Linux-specific (that file had to be
split off into a Linux and Darwin version, although I think that
could be cleaned up a fair amount; I am unclear as to why some of
that split exists, although it might be required for the server
support).  I did steal a few routines from lclient, but a large
number of those routines were extrememly non-portable so I couldn''t
use those files as-is; I didn''t want to have a huge mess of
ifdef''s
in there so I just stole the few routines I needed.

However ... I think the changes for the Mac client are pretty
self-contained; the vast majority of the "real" magic is in libcfs
and lvnode.  From my experience in tracking master, very few changes
affect me.

(You can look at this yourself once we get the repository stuff
straightened out; I think we''re almost there).

--Ken