thr3ads.net - Lustre discuss - [Lustre-discuss] Unable to build kernel-ib when building lustre from source code [Feb 2010]

If this information is useful, please help other people find it:
Share via:

neutron

2010-Feb-01 02:12 UTC

[Lustre-discuss] Unable to build kernel-ib when building lustre from source code

Hi all,

I have installed Lustre-1.8.1.1  from pre-built rpm packages and it
works. Now I want to build Lustre 1.8.1.1 from source on RHEL5
x86_64.  Since I want to learn Lustre( and eventually make some
modifications in lustre),   I choose to build Lustre from source code.


I followed the steps at "Installing Lustre with a Third-Party Network
Stack" from "lustre manual 1.8".
Here is what I do.  At my ~/.rpmmacros,  I set my rpm working dir to
be ~/rpmbuild

(1) Install "kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1.src.rpm".
The manual says to use "kernel-lustre-source-xxxx.rpm", but I
couldn''t
find that file at Sun''s download site.
This src.rpm is installed to ~/rpmbuild/SOURCE

(2) At ~/rpmbuild/SPECS,  Run "rpmbuild -bp --target=x86_64 
./kernel-2.6.spec"
The kernel src tree is at
"~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64"

(3) In that dir,  I use the config file of currently running kernel, and then:
	$ make oldconfig || make menuconfig
	$ make include/asm
	$ make include/linux/version.h
	$ make SUBDIRS=scripts
	$ make;  make modules_install; make install

This should be identical to "make rpm" and then install the
kernel-xxx.rpm and kernel-xxx.src.rpm.

(4) Reboot into the new lustre kernel
"uname -r" gives  "2.6.18-128.7.1-lustre-1.8.1-customer".
I have verified that
"/lib/modules/2.6.18-128.7.1-lustre-1.8.1-customer/build" point to
"~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64"


(5) Install the
"kernel-ib-devel-1.4.2-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm".
The files are installed to "/usr/src/ofa_kernel-1.4.2".
 At that place, I run
	$ /configure --with-core-mod --with-ipoib-mod --with-mthca-mod
--with-mlx4_core-mod --with-mlx4_inf-mod
	$ make

Then I see lots of errors like:
	/usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/fs.h:9:
error: redefinition of ?inc_nlink?
include/linux/fs.h:1255: error: previous definition of ?inc_nlink? was here
In file included from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:44,
                 from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38,
                 from /usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49:
/usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/mm.h:17:
error: redefinition of ?is_vmalloc_addr?
include/linux/mm.h:332: error: previous definition of ?is_vmalloc_addr? was here
In file included from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38,
                 from /usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49:
/usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h: In function
?ib_dma_mapping_error?:
/usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:1593: warning:
passing argument 1 of ?dma_mapping_error? makes integer from pointer
without a cast


I googled the error, and someone said to use OFED-1.5.
But I tried with OFED-1.4.2, OFED-1.5, and OFED-1.5.1 daily tar ball
OFED-1.5.1-20100131-0600,
and always see the same error "redefinition of ........"   when
I''m
compiling the kernel-ib.

Am I wrong with the way to install kernel-ib or kernel-lustre?  How
can I make lustre work with IB support?


(6) Build lustre succeeds without errors.
I untared "lustre-1.8.1.1.tar.gz", then
        $ ./configure
--with-linux=~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64
	$ make
	$ make install

Looks both lustre kernel modules and luster userland tools/library are
installed correctly.
Is this a right way to install lustre?

Thanks!



-Neutron

Atul Vidwansa

2010-Feb-01 03:44 UTC

head link

[Lustre-discuss] Unable to build kernel-ib when building lustre from source code

Sometime back I posted instruction on building Lustre from sources with 
OFED on my blog at http://blogs.sun.com/atulvid/ . Have a look, it may 
be useful to you.

Cheers,
_Atul

neutron wrote:> Hi all,
>
> I have installed Lustre-1.8.1.1  from pre-built rpm packages and it
> works. Now I want to build Lustre 1.8.1.1 from source on RHEL5
> x86_64.  Since I want to learn Lustre( and eventually make some
> modifications in lustre),   I choose to build Lustre from source code.
>
>
> I followed the steps at "Installing Lustre with a Third-Party Network
> Stack" from "lustre manual 1.8".
> Here is what I do.  At my ~/.rpmmacros,  I set my rpm working dir to
> be ~/rpmbuild
>
> (1) Install
"kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1.src.rpm".
> The manual says to use "kernel-lustre-source-xxxx.rpm", but I
couldn''t
> find that file at Sun''s download site.
> This src.rpm is installed to ~/rpmbuild/SOURCE
>
> (2) At ~/rpmbuild/SPECS,  Run "rpmbuild -bp --target=x86_64 
./kernel-2.6.spec"
> The kernel src tree is at
> "~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64"
>
> (3) In that dir,  I use the config file of currently running kernel, and
then:
> 	$ make oldconfig || make menuconfig
> 	$ make include/asm
> 	$ make include/linux/version.h
> 	$ make SUBDIRS=scripts
> 	$ make;  make modules_install; make install
>
> This should be identical to "make rpm" and then install the
> kernel-xxx.rpm and kernel-xxx.src.rpm.
>
> (4) Reboot into the new lustre kernel
> "uname -r" gives 
"2.6.18-128.7.1-lustre-1.8.1-customer".
> I have verified that
> "/lib/modules/2.6.18-128.7.1-lustre-1.8.1-customer/build" point
to
> "~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64"
>
>
> (5) Install the
>
"kernel-ib-devel-1.4.2-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm".
> The files are installed to "/usr/src/ofa_kernel-1.4.2".
>  At that place, I run
> 	$ /configure --with-core-mod --with-ipoib-mod --with-mthca-mod
> --with-mlx4_core-mod --with-mlx4_inf-mod
> 	$ make
>
> Then I see lots of errors like:
> 
/usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/fs.h:9:
> error: redefinition of ?inc_nlink?
> include/linux/fs.h:1255: error: previous definition of ?inc_nlink? was here
> In file included from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:44,
>                  from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38,
>                  from
/usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49:
>
/usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/mm.h:17:
> error: redefinition of ?is_vmalloc_addr?
> include/linux/mm.h:332: error: previous definition of ?is_vmalloc_addr? was
here
> In file included from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38,
>                  from
/usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49:
> /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h: In function
> ?ib_dma_mapping_error?:
> /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:1593: warning:
> passing argument 1 of ?dma_mapping_error? makes integer from pointer
> without a cast
>
>
> I googled the error, and someone said to use OFED-1.5.
> But I tried with OFED-1.4.2, OFED-1.5, and OFED-1.5.1 daily tar ball
> OFED-1.5.1-20100131-0600,
> and always see the same error "redefinition of ........"   when
I''m
> compiling the kernel-ib.
>
> Am I wrong with the way to install kernel-ib or kernel-lustre?  How
> can I make lustre work with IB support?
>
>
> (6) Build lustre succeeds without errors.
> I untared "lustre-1.8.1.1.tar.gz", then
>         $ ./configure
> --with-linux=~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64
> 	$ make
> 	$ make install
>
> Looks both lustre kernel modules and luster userland tools/library are
> installed correctly.
> Is this a right way to install lustre?
>
> Thanks!
>
>
>
> -Neutron
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>

Brian J. Murrell

2010-Feb-01 16:57 UTC

head link

[Lustre-discuss] Unable to build kernel-ib when building lustre from source code

On Sun, 2010-01-31 at 21:12 -0500, neutron wrote: > Hi all,
Hi,
> (5) Install the
>
"kernel-ib-devel-1.4.2-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm".
> The files are installed to "/usr/src/ofa_kernel-1.4.2".
>  At that place, I run
> 	$ /configure --with-core-mod --with-ipoib-mod --with-mthca-mod
> --with-mlx4_core-mod --with-mlx4_inf-mod
> 	$ make
> 
> Then I see lots of errors like:
> 
/usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/fs.h:9:
> error: redefinition of ?inc_nlink?
> include/linux/fs.h:1255: error: previous definition of ?inc_nlink? was here
> In file included from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:44,
>                  from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38,
>                  from
/usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49:
>
/usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/mm.h:17:
> error: redefinition of ?is_vmalloc_addr?
> include/linux/mm.h:332: error: previous definition of ?is_vmalloc_addr? was
here
> In file included from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38,
>                  from
/usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49:
> /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h: In function
> ?ib_dma_mapping_error?:
> /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:1593: warning:
> passing argument 1 of ?dma_mapping_error? makes integer from pointer
> without a cast
These are all OFED building errors, and have nothing to do with Lustre.
You *might* get help here in resolving those, but you would probably
have much more luck if you go to the OFED community and ask them
directly.

Having said that, I do notice from the OFED 1.4.2 announcement and
release notes
(http://lists.openfabrics.org/pipermail/ewg/2009-August/013731.html)
that the newest RHEL5 kernel supported is:

- RedHat EL5 up3: 2.6.18-128.el5

Which is older than the kernel you are trying to build with:
2.6.18_128.7.1.el5
> I googled the error, and someone said to use OFED-1.5.
Probably because of the kernel being newer than OFED 1.4.2 supports and
therefore not supported until OFED 1.5.
> But I tried with OFED-1.4.2, OFED-1.5, and OFED-1.5.1 daily tar ball
> OFED-1.5.1-20100131-0600,
> and always see the same error "redefinition of ........"   when
I''m
> compiling the kernel-ib.
Again, this appears to be strictly an OFED issue and the OFED list might
get you more help.
> Am I wrong with the way to install kernel-ib or kernel-lustre?  How
> can I make lustre work with IB support?
You are not even that far (making lustre work with IB support) yet.  You
are still trying to build OFED which has to be done before building
Lustre.

Cheers,
b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100201/4c938950/attachment.bin

neutron

2010-Feb-02 16:48 UTC

head link

[Lustre-discuss] Unable to build kernel-ib when building lustre from source code

Hi Atul, thanks a lot!   Now I can build lustre, ofed.

When I run mkfs.lustre, a strange thing happens:

--------------
[i3:~/lustre/scripts]$sudo /usr/sbin/mkfs.lustre --fsname=lustre
--reformat --mdt --mgs --device-size=100000  /dev/os/lustre
Password:

   Permanent disk data:
Target:     lustre-MDTffff
Index:      unassigned
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x75
              (MDT MGS needs_index first_time update )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters: mdt.group_upcall=/usr/sbin/l_getgroups

2 6 18
formatting backing filesystem ldiskfs on /dev/os/lustre
        target name  lustre-MDTffff
        4k blocks     25000
        options        -i 4096 -I 512 -q -O dir_index,extents,uninit_groups -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre-MDTffff  -i 4096 -I 512 -q -O
dir_index,extents,uninit_groups -F /dev/os/lustre 25000
   sh: mke2fs: command not found

mkfs.lustre FATAL: Unable to build fs /dev/os/lustre (32512)

mkfs.lustre FATAL: mkfs failed 32512


---------------
It appears as if mkfs.lustre cannot find mke2fs.  I have installed the
e2fsprogs.rpm.  "mke2fs" is in dir "/sbin", and
it''s already included
in the PATH.

[i3:~/lustre/lustre-1.8.1]$rpm -q e2fsprogs
e2fsprogs-1.41.6.sun1-0redhat

[i3:~/lustre/lustre-1.8.1]$echo $PATH
/sbin:/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/local/ofed/bin:/usr/local/ofed/sbin:/usr/kerberos/bin:/usr/local/bin



On Sun, Jan 31, 2010 at 10:44 PM, Atul Vidwansa <Atul.Vidwansa at sun.com>
wrote:> Sometime back I posted instruction on building Lustre from sources with
OFED
> on my blog at http://blogs.sun.com/atulvid/ . Have a look, it may be useful
> to you.
>
> Cheers,
> _Atul
>
> neutron wrote:
>>
>> Hi all,
>>
>> I have installed Lustre-1.8.1.1 ?from pre-built rpm packages and it
>> works. Now I want to build Lustre 1.8.1.1 from source on RHEL5
>> x86_64. ?Since I want to learn Lustre( and eventually make some
>> modifications in lustre), ? I choose to build Lustre from source code.
>>
>>
>> I followed the steps at "Installing Lustre with a Third-Party
Network
>> Stack" from "lustre manual 1.8".
>> Here is what I do. ?At my ~/.rpmmacros, ?I set my rpm working dir to
>> be ~/rpmbuild
>>
>> (1) Install
"kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1.src.rpm".
>> The manual says to use "kernel-lustre-source-xxxx.rpm", but I
couldn''t
>> find that file at Sun''s download site.
>> This src.rpm is installed to ~/rpmbuild/SOURCE
>>
>> (2) At ~/rpmbuild/SPECS, ?Run "rpmbuild -bp --target=x86_64
>> ?./kernel-2.6.spec"
>> The kernel src tree is at
>> "~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64"
>>
>> (3) In that dir, ?I use the config file of currently running kernel,
and
>> then:
>> ? ? ? ?$ make oldconfig || make menuconfig
>> ? ? ? ?$ make include/asm
>> ? ? ? ?$ make include/linux/version.h
>> ? ? ? ?$ make SUBDIRS=scripts
>> ? ? ? ?$ make; ?make modules_install; make install
>>
>> This should be identical to "make rpm" and then install the
>> kernel-xxx.rpm and kernel-xxx.src.rpm.
>>
>> (4) Reboot into the new lustre kernel
>> "uname -r" gives
?"2.6.18-128.7.1-lustre-1.8.1-customer".
>> I have verified that
>> "/lib/modules/2.6.18-128.7.1-lustre-1.8.1-customer/build"
point to
>> "~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64"
>>
>>
>> (5) Install the
>>
"kernel-ib-devel-1.4.2-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm".
>> The files are installed to "/usr/src/ofa_kernel-1.4.2".
>> ?At that place, I run
>> ? ? ? ?$ /configure --with-core-mod --with-ipoib-mod --with-mthca-mod
>> --with-mlx4_core-mod --with-mlx4_inf-mod
>> ? ? ? ?$ make
>>
>> Then I see lots of errors like:
>>
>>
?/usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/fs.h:9:
>> error: redefinition of ?inc_nlink?
>> include/linux/fs.h:1255: error: previous definition of ?inc_nlink? was
>> here
>> In file included from
>> /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:44,
>> ? ? ? ? ? ? ? ? from
/usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38,
>> ? ? ? ? ? ? ? ? from
>> /usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49:
>>
>>
/usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/mm.h:17:
>> error: redefinition of ?is_vmalloc_addr?
>> include/linux/mm.h:332: error: previous definition of ?is_vmalloc_addr?
>> was here
>> In file included from
>> /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38,
>> ? ? ? ? ? ? ? ? from
>> /usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49:
>> /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h: In function
>> ?ib_dma_mapping_error?:
>> /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:1593: warning:
>> passing argument 1 of ?dma_mapping_error? makes integer from pointer
>> without a cast
>>
>>
>> I googled the error, and someone said to use OFED-1.5.
>> But I tried with OFED-1.4.2, OFED-1.5, and OFED-1.5.1 daily tar ball
>> OFED-1.5.1-20100131-0600,
>> and always see the same error "redefinition of ........" ?
when I''m
>> compiling the kernel-ib.
>>
>> Am I wrong with the way to install kernel-ib or kernel-lustre? ?How
>> can I make lustre work with IB support?
>>
>>
>> (6) Build lustre succeeds without errors.
>> I untared "lustre-1.8.1.1.tar.gz", then
>> ? ? ? ?$ ./configure
>> --with-linux=~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64
>> ? ? ? ?$ make
>> ? ? ? ?$ make install
>>
>> Looks both lustre kernel modules and luster userland tools/library are
>> installed correctly.
>> Is this a right way to install lustre?
>>
>> Thanks!
>>
>>
>>
>> -Neutron
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>
>

Lustre discuss - Feb 2010 - Unable to build kernel-ib when building lustre from source code

[Lustre-discuss] Unable to build kernel-ib when building lustre from source code

[Lustre-discuss] Unable to build kernel-ib when building lustre from source code

[Lustre-discuss] Unable to build kernel-ib when building lustre from source code

[Lustre-discuss] Unable to build kernel-ib when building lustre from source code