Lustre discuss - Jan 2007 - Questions on building & packaging Lustre -1.5.95 to run on different target machine

Skipped content of type multipart/alternative-------------- next part
--------------
A non-text attachment was scrubbed...
Name: consoleOutput
Type: application/octet-stream
Size: 4156 bytes
Desc: consoleOutput
Url :
http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20070109/a88b06ce/consoleOutput-0001.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config.log
Type: application/octet-stream
Size: 134720 bytes
Desc: config.log
Url :
http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20070109/a88b06ce/config-0001.obj

AFAIK, we don''t ever cross-compile ourselves, so it''s quite
possible
there are problems
with our configure scripts.
Andy, this is potentially another area to add to our testing coverage.

Snider, Tim wrote:
> I''m attempting to cross compile, build, and collect files to
install
> Lustre and run on a separate machine.
> According to the documentation and web pages Lustre can be installed 
> in an alternate directory path according to the web pages and 
> configure using the --prefix, --exec-prefix, --with-lustre options 
> when configure is run.
> For me the configure script seems to be ignoring those options.
>  
> 2 unique problems occur depending on if I build as a normal user or as 
> root.
>  
> Environment settings for both user and root builds are:
>    LINUX=/home/tsnider/linux-2.6.18.3
>    GCC_EXEC_PREFIX=/home/tsnider/tsLustre/lustre-1.5.95
>    CFLAGS="-I/cross/armv5l-linux/include -g -O2"
>    GCC=/cross/armv5l-linux/bin:/cross/libexec/gcc/armv5l-linux/3.4.4
>    EXEC_PREFIX=/home/tsnider/tsLustre/lustre-1.5.95
>    CC=/cross/armv5l-linux/bin:/cross/libexec/gcc/armv5l-linux/3.4.4
>    LUSTRE=/home/tsnider/tsLustre/lustre-1.5.95
>  
> A common configuration command is also used:
>  sh configure --host=armv5l-linux --target=armv5l-linux 
> --with-linux=/home/tsnider/linux-2.6.18.3  --disable-l iblustre 
> --prefix=/home/tsnider/lustre2/installLustre/ --  
> --exec-prefix=/home/tsnider/lustre2/installLustre//
>  
> According to my understanding the --with-linux, --prefix, and 
> --exec-prefix should cause Lustre executables/files
> to be installed in the specified alternate path.
>  
> "Normal user" make issue:
>     As a normal user "make install" tries to copy files to the
system
> path /lib/modules/2.6.18.3......
>  Make errors out (as expected) since a normal user can''t modify
system
> paths/files.
>     I assumed that one of the alternate  paths would be used since 
> they were specified during configure, but they weren''t.
>  
>     There are 2 main possibilities for sources of the problem:
>             A. pilot error (probably)
>             B. Configure (& associated scripts) ignore the alternate 
> path options.
>  
>    "make install" output as user:
>      make[3]: Entering directory 
> `/home/tsnider/tsLustre/lustre-1.5.95/lnet/libcfs/linux''
>      make[4]: Entering directory 
> `/home/tsnider/tsLustre/lustre-1.5.95/lnet/libcfs/linux''
>      make[4]: Nothing to be done for `install-exec-am''.
>      make[4]: Nothing to be done for `install-data-am''.
>      make[4]: Leaving directory 
> `/home/tsnider/tsLustre/lustre-1.5.95/lnet/libcfs/linux''
>      make[3]: Leaving directory 
> `/home/tsnider/tsLustre/lustre-1.5.95/lnet/libcfs/linux''
>      make[3]: Entering directory 
> `/home/tsnider/tsLustre/lustre-1.5.95/lnet/libcfs''
>      make[4]: Entering directory 
> `/home/tsnider/tsLustre/lustre-1.5.95/lnet/libcfs''
>      make[4]: Nothing to be done for `install-exec-am''.
>      /bin/sh ../../mkinstalldirs
>      /bin/sh ../../mkinstalldirs /lib/modules/2.6.18.3/kernel/net/lustre
>      mkdir -p -- /lib/modules/2.6.18.3/kernel/net/lustre
>      mkdir: cannot create directory 
> `/lib/modules/2.6.18.3/kernel/net/lustre'': Permission denied
>      make[4]: *** [install-modulenetDATA] Error 1
>      make[4]: Leaving directory 
> `/home/tsnider/tsLustre/lustre-1.5.95/lnet/libcfs''
>      make[3]: *** [install-am] Error 2
>  
>  Why doesn''t mkinstalldirs redirect the install path to the one 
> specified by the --with-linux, --prefix, or --exec-prefix
>  option that was given during configure?  Am I missing something or do 
> the configure scripts need to be investigated?
>  
> "Root" make issue:
>  
>  I understand from the manual and read me files that RPMs can not be 
> made for distros other than redhat or sles.
>  "make install" will build lustre. It appears that installation
path
> directives are ignored. The attached file
>  shows that makinstalldirs behaves inconsistently. Sometimes it 
> installs in the specified path and other times it
>  installs in the system path.
>    /bin/sh ../../../mkinstalldirs /lib/modules/2.6.18.3/kernel/net/lustre
>    /bin/sh ../../mkinstalldirs
>    /bin/sh ../../mkinstalldirs
>    /bin/sh ../../mkinstalldirs /home/tsnider/lustre2/installLustre//lib
>    /bin/sh ../../mkinstalldirs /lib/modules/2.6.18.3/kernel/fs/lustre
>  For cross compilation and installation on another machine files 
> should be in a common place so they can
>  be packaged, in some manner, and sent to the target machine.
>  I''d like to know how to correctly specify options to do this. Or
do
> the auto config/make scripts need
>  work in order for this to work?
>  
>     As root "make install" Details of errors are below.
>  
> Other minor things:
>  1. files have to modified during build, the patches don''t seem to
be
> entirely complete.
>  
>  2. Configure spits out the following warning:
>    configure: WARNING: If you wanted to set the --build type,
don''t
> use --host.
>       If a cross compiler is detected then cross compile mode will be 
> used.
>  3.  a few warnings appear for 2.6 port / implemnentation stuff:
>    /home/tsnider/lustre2/lustre-1.5.95/lustre/obdecho/echo_client.c:650:2: 
> warning: #warning "echo_client_ubrw() needs to be ported on 2.6
yet"
>   
/home/tsnider/lustre2/lustre-1.5.95/lustre/obdfilter/lproc_obdfilter.c:392:2:
> warning: #warning "port on 2.6 -bzzz"
>  
> Any help someone can give is appreciated. I don''t know if
it''s
> possible to do what I''d like to do. The attached file contains the
> console output from configure and build commands.
>  
> Thanks,
> Timothy Snider
> Storage Architect
> Strategic Planning, Technology and Architecture
>
> LSI Logic Corporation
> 3718 North Rock Road
> Wichita, KS 67226
> (316) 636-8736
> _tim.snider@lsi.com <mailto:tim.snider@lsi.com>_
>
>  
> ------------------------------------------------------------------------
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss@clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss

That''s what I was beginning to think after trying to get this to work
over the last 4 weeks. Is there any documentation that describes all the
files produced during make and where they go? This would be better than
trying to wade thru all the configuration and make files/scripts.
Thanks for the confirmation.

-----Original Message-----
From: Nathaniel Rutman [mailto:nathan@clusterfs.com] 
Sent: Tuesday, January 09, 2007 4:42 PM
To: Snider, Tim
Cc: lustre-discuss@clusterfs.com; Andy Rudoff
Subject: Re: [Lustre-discuss] Questions on building & packaging Lustre
-1.5.95 to run on different target machine

AFAIK, we don''t ever cross-compile ourselves, so it''s quite
possible
there are problems with our configure scripts.
Andy, this is potentially another area to add to our testing coverage.

Snider, Tim wrote:
> I''m attempting to cross compile, build, and collect files to
install
> Lustre and run on a separate machine.
> According to the documentation and web pages Lustre can be installed 
> in an alternate directory path according to the web pages and 
> configure using the --prefix, --exec-prefix, --with-lustre options 
> when configure is run.
> For me the configure script seems to be ignoring those options.
>  
> 2 unique problems occur depending on if I build as a normal user or as
> root.
>  
> Environment settings for both user and root builds are:
>    LINUX=/home/tsnider/linux-2.6.18.3
>    GCC_EXEC_PREFIX=/home/tsnider/tsLustre/lustre-1.5.95
>    CFLAGS="-I/cross/armv5l-linux/include -g -O2"
>    GCC=/cross/armv5l-linux/bin:/cross/libexec/gcc/armv5l-linux/3.4.4
>    EXEC_PREFIX=/home/tsnider/tsLustre/lustre-1.5.95
>    CC=/cross/armv5l-linux/bin:/cross/libexec/gcc/armv5l-linux/3.4.4
>    LUSTRE=/home/tsnider/tsLustre/lustre-1.5.95
>  
> A common configuration command is also used:
>  sh configure --host=armv5l-linux --target=armv5l-linux
> --with-linux=/home/tsnider/linux-2.6.18.3  --disable-l iblustre 
> --prefix=/home/tsnider/lustre2/installLustre/ -- 
> --exec-prefix=/home/tsnider/lustre2/installLustre//
>  
> According to my understanding the --with-linux, --prefix, and 
> --exec-prefix should cause Lustre executables/files to be installed in
> the specified alternate path.
>  
> "Normal user" make issue:
>     As a normal user "make install" tries to copy files to the
system
> path /lib/modules/2.6.18.3......
>  Make errors out (as expected) since a normal user can''t modify
system
> paths/files.
>     I assumed that one of the alternate  paths would be used since 
> they were specified during configure, but they weren''t.
>  
>     There are 2 main possibilities for sources of the problem:
>             A. pilot error (probably)
>             B. Configure (& associated scripts) ignore the alternate 
> path options.
>  
>    "make install" output as user:
>      make[3]: Entering directory
> `/home/tsnider/tsLustre/lustre-1.5.95/lnet/libcfs/linux''
>      make[4]: Entering directory
> `/home/tsnider/tsLustre/lustre-1.5.95/lnet/libcfs/linux''
>      make[4]: Nothing to be done for `install-exec-am''.
>      make[4]: Nothing to be done for `install-data-am''.
>      make[4]: Leaving directory
> `/home/tsnider/tsLustre/lustre-1.5.95/lnet/libcfs/linux''
>      make[3]: Leaving directory
> `/home/tsnider/tsLustre/lustre-1.5.95/lnet/libcfs/linux''
>      make[3]: Entering directory
> `/home/tsnider/tsLustre/lustre-1.5.95/lnet/libcfs''
>      make[4]: Entering directory
> `/home/tsnider/tsLustre/lustre-1.5.95/lnet/libcfs''
>      make[4]: Nothing to be done for `install-exec-am''.
>      /bin/sh ../../mkinstalldirs
>      /bin/sh ../../mkinstalldirs
/lib/modules/2.6.18.3/kernel/net/lustre
>      mkdir -p -- /lib/modules/2.6.18.3/kernel/net/lustre
>      mkdir: cannot create directory
> `/lib/modules/2.6.18.3/kernel/net/lustre'': Permission denied
>      make[4]: *** [install-modulenetDATA] Error 1
>      make[4]: Leaving directory
> `/home/tsnider/tsLustre/lustre-1.5.95/lnet/libcfs''
>      make[3]: *** [install-am] Error 2
>  
>  Why doesn''t mkinstalldirs redirect the install path to the one 
> specified by the --with-linux, --prefix, or --exec-prefix  option that
> was given during configure?  Am I missing something or do the 
> configure scripts need to be investigated?
>  
> "Root" make issue:
>  
>  I understand from the manual and read me files that RPMs can not be 
> made for distros other than redhat or sles.
>  "make install" will build lustre. It appears that installation
path
> directives are ignored. The attached file  shows that makinstalldirs 
> behaves inconsistently. Sometimes it installs in the specified path 
> and other times it  installs in the system path.
>    /bin/sh ../../../mkinstalldirs
/lib/modules/2.6.18.3/kernel/net/lustre
>    /bin/sh ../../mkinstalldirs
>    /bin/sh ../../mkinstalldirs
>    /bin/sh ../../mkinstalldirs
/home/tsnider/lustre2/installLustre//lib
>    /bin/sh ../../mkinstalldirs /lib/modules/2.6.18.3/kernel/fs/lustre
>  For cross compilation and installation on another machine files 
> should be in a common place so they can  be packaged, in some manner, 
> and sent to the target machine.
>  I''d like to know how to correctly specify options to do this. Or
do
> the auto config/make scripts need  work in order for this to work?
>  
>     As root "make install" Details of errors are below.
>  
> Other minor things:
>  1. files have to modified during build, the patches don''t seem to
be
> entirely complete.
>  
>  2. Configure spits out the following warning:
>    configure: WARNING: If you wanted to set the --build type,
don''t
> use --host.
>       If a cross compiler is detected then cross compile mode will be 
> used.
>  3.  a few warnings appear for 2.6 port / implemnentation stuff:
>
/home/tsnider/lustre2/lustre-1.5.95/lustre/obdecho/echo_client.c:650:2:
> warning: #warning "echo_client_ubrw() needs to be ported on 2.6
yet"
>
/home/tsnider/lustre2/lustre-1.5.95/lustre/obdfilter/lproc_obdfilter.c:3
92:2: 
> warning: #warning "port on 2.6 -bzzz"
>  
> Any help someone can give is appreciated. I don''t know if
it''s
> possible to do what I''d like to do. The attached file contains the
> console output from configure and build commands.
>  
> Thanks,
> Timothy Snider
> Storage Architect
> Strategic Planning, Technology and Architecture
>
> LSI Logic Corporation
> 3718 North Rock Road
> Wichita, KS 67226
> (316) 636-8736
> _tim.snider@lsi.com <mailto:tim.snider@lsi.com>_
>
>  
> ----------------------------------------------------------------------
> --
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss@clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss

Snider, Tim wrote:
> I''m attempting to cross compile, build, and collect files to
install
> Lustre and run on a separate machine.
> According to the documentation and web pages Lustre can be installed in 
> an alternate directory path according to the web pages and configure 
> using the --prefix, --exec-prefix, --with-lustre options when configure 
> is run.
> For me the configure script seems to be ignoring those options.
Sorry about the delay in responding, on the road.  hopefully I can help. 
Is this really a true cross-compile? (different CPU arch, requires 
different compilers?) If so, I''d say it''s
highly unlikely to work, as we have never supported other arches and we 
don''t cross-compile. (Unless you are a Cray
XT3)
>  
> 2 unique problems occur depending on if I build as a normal user or as
root.
>  
> Environment settings for both user and root builds are:
>    LINUX=/home/tsnider/linux-2.6.18.3
>    GCC_EXEC_PREFIX=/home/tsnider/tsLustre/lustre-1.5.95
>    CFLAGS="-I/cross/armv5l-linux/include -g -O2"
>    GCC=/cross/armv5l-linux/bin:/cross/libexec/gcc/armv5l-linux/3.4.4
>    EXEC_PREFIX=/home/tsnider/tsLustre/lustre-1.5.95
>    CC=/cross/armv5l-linux/bin:/cross/libexec/gcc/armv5l-linux/3.4.4
>    LUSTRE=/home/tsnider/tsLustre/lustre-1.5.95
>  
> A common configuration command is also used:
>  sh configure --host=armv5l-linux --target=armv5l-linux 
> --with-linux=/home/tsnider/linux-2.6.18.3  --disable-l iblustre 
> --prefix=/home/tsnider/lustre2/installLustre/ --  
> --exec-prefix=/home/tsnider/lustre2/installLustre//
In your config.log I see:
configure:1447: checking build system type
configure:1465: result: x86_64-redhat-linux-gnu

That doesn''t look like a cross-compile. You should not need or
use the --host and --target options.
>  
> According to my understanding the --with-linux, --prefix, and 
> --exec-prefix should cause Lustre executables/files
> to be installed in the specified alternate path.

--with-linux indicates where the kernel source is (needed to compile 
modules) I don''t think it changes where the bits are installed.
>  
> "Normal user" make issue:
>     As a normal user "make install" tries to copy files to the
system
> path /lib/modules/2.6.18.3......
>  Make errors out (as expected) since a normal user can''t modify
system
> paths/files.
>     I assumed that one of the alternate  paths would be used since they 
> were specified during configure, but they weren''t.
>  
>     There are 2 main possibilities for sources of the problem:
>             A. pilot error (probably)
>             B. Configure (& associated scripts) ignore the alternate 
> path options.
	C. the GNU configure goop (--prefix,etc ) applies to user-space 
programs, afaik anytime we build kernel modules we use the Linux kernel 
build system, which is called by our Makefiles, and thus kernel modules 
are installed in a directory that matches the kernel source suppled.
( as an aside, i dunno how you make Linux put kernel modules anyplace 
_other than /lib/modules..)
Was the kernel source you pointed to set up for the cross-compile?
>  
>  Why doesn''t mkinstalldirs redirect the install path to the one 
> specified by the --with-linux, --prefix, or --exec-prefix
>  option that was given during configure?  Am I missing something or do 
> the configure scripts need to be investigated?
>  
> "Root" make issue:
>  
>  I understand from the manual and read me files that RPMs can not be 
> made for distros other than redhat or sles.
What distro do you need? We build on anything that is remotely 
redhat-ish (Centos, Fedora) If your build system is
x86_64-redhat-linux-gnu, ''make rpms'' should Just Work.
>  "make install" will build lustre. It appears that installation
path
> directives are ignored. The attached file
>  shows that makinstalldirs behaves inconsistently. Sometimes it installs 
> in the specified path and other times it
>  installs in the system path.
>    /bin/sh ../../../mkinstalldirs /lib/modules/2.6.18.3/kernel/net/lustre
>    /bin/sh ../../mkinstalldirs
>    /bin/sh ../../mkinstalldirs
>    /bin/sh ../../mkinstalldirs /home/tsnider/lustre2/installLustre//lib
>    /bin/sh ../../mkinstalldirs /lib/modules/2.6.18.3/kernel/fs/lustre
>  For cross compilation and installation on another machine files should 
> be in a common place so they can
>  be packaged, in some manner, and sent to the target machine.
>  I''d like to know how to correctly specify options to do this. Or
do the
> auto config/make scripts need
>  work in order for this to work?
The best way to do this with current Lustre is ''make
rpms''
>  
>     As root "make install" Details of errors are below.
>  
> Other minor things:
>  1. files have to modified during build, the patches don''t seem to
be
> entirely complete.
>  
That''s quite possible with these 1.6 betas, please post specific errors
to lustre-devel and the 1.6 beta team will catch them.
>  2. Configure spits out the following warning:
>    configure: WARNING: If you wanted to set the --build type,
don''t use
> --host.
>       If a cross compiler is detected then cross compile mode will be used.
>  3.  a few warnings appear for 2.6 port / implemnentation stuff:
>    /home/tsnider/lustre2/lustre-1.5.95/lustre/obdecho/echo_client.c:650:2: 
> warning: #warning "echo_client_ubrw() needs to be ported on 2.6
yet"
>   
/home/tsnider/lustre2/lustre-1.5.95/lustre/obdfilter/lproc_obdfilter.c:392:2:
> warning: #warning "port on 2.6 -bzzz"
I don''t think these errors are meaningful. echo_client is used for 
testing only.
>  
> Any help someone can give is appreciated. I don''t know if
it''s possible
> to do what I''d like to do. The attached file contains the console
output
> from configure and build commands.
Key questions - what CPU arch are you attempting to build for?
Best Regards
cliffw
>  
> Thanks,
> Timothy Snider
> Storage Architect
> Strategic Planning, Technology and Architecture
> 
> LSI Logic Corporation
> 3718 North Rock Road
> Wichita, KS 67226
> (316) 636-8736
> _tim.snider@lsi.com <mailto:tim.snider@lsi.com>_
> 
>  
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss@clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss