neutron
2010-Feb-01 02:12 UTC
[Lustre-discuss] Unable to build kernel-ib when building lustre from source code
Hi all, I have installed Lustre-1.8.1.1 from pre-built rpm packages and it works. Now I want to build Lustre 1.8.1.1 from source on RHEL5 x86_64. Since I want to learn Lustre( and eventually make some modifications in lustre), I choose to build Lustre from source code. I followed the steps at "Installing Lustre with a Third-Party Network Stack" from "lustre manual 1.8". Here is what I do. At my ~/.rpmmacros, I set my rpm working dir to be ~/rpmbuild (1) Install "kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1.src.rpm". The manual says to use "kernel-lustre-source-xxxx.rpm", but I couldn''t find that file at Sun''s download site. This src.rpm is installed to ~/rpmbuild/SOURCE (2) At ~/rpmbuild/SPECS, Run "rpmbuild -bp --target=x86_64 ./kernel-2.6.spec" The kernel src tree is at "~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64" (3) In that dir, I use the config file of currently running kernel, and then: $ make oldconfig || make menuconfig $ make include/asm $ make include/linux/version.h $ make SUBDIRS=scripts $ make; make modules_install; make install This should be identical to "make rpm" and then install the kernel-xxx.rpm and kernel-xxx.src.rpm. (4) Reboot into the new lustre kernel "uname -r" gives "2.6.18-128.7.1-lustre-1.8.1-customer". I have verified that "/lib/modules/2.6.18-128.7.1-lustre-1.8.1-customer/build" point to "~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64" (5) Install the "kernel-ib-devel-1.4.2-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm". The files are installed to "/usr/src/ofa_kernel-1.4.2". At that place, I run $ /configure --with-core-mod --with-ipoib-mod --with-mthca-mod --with-mlx4_core-mod --with-mlx4_inf-mod $ make Then I see lots of errors like: /usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/fs.h:9: error: redefinition of ?inc_nlink? include/linux/fs.h:1255: error: previous definition of ?inc_nlink? was here In file included from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:44, from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38, from /usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49: /usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/mm.h:17: error: redefinition of ?is_vmalloc_addr? include/linux/mm.h:332: error: previous definition of ?is_vmalloc_addr? was here In file included from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38, from /usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49: /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h: In function ?ib_dma_mapping_error?: /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:1593: warning: passing argument 1 of ?dma_mapping_error? makes integer from pointer without a cast I googled the error, and someone said to use OFED-1.5. But I tried with OFED-1.4.2, OFED-1.5, and OFED-1.5.1 daily tar ball OFED-1.5.1-20100131-0600, and always see the same error "redefinition of ........" when I''m compiling the kernel-ib. Am I wrong with the way to install kernel-ib or kernel-lustre? How can I make lustre work with IB support? (6) Build lustre succeeds without errors. I untared "lustre-1.8.1.1.tar.gz", then $ ./configure --with-linux=~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64 $ make $ make install Looks both lustre kernel modules and luster userland tools/library are installed correctly. Is this a right way to install lustre? Thanks! -Neutron
Atul Vidwansa
2010-Feb-01 03:44 UTC
[Lustre-discuss] Unable to build kernel-ib when building lustre from source code
Sometime back I posted instruction on building Lustre from sources with OFED on my blog at http://blogs.sun.com/atulvid/ . Have a look, it may be useful to you. Cheers, _Atul neutron wrote:> Hi all, > > I have installed Lustre-1.8.1.1 from pre-built rpm packages and it > works. Now I want to build Lustre 1.8.1.1 from source on RHEL5 > x86_64. Since I want to learn Lustre( and eventually make some > modifications in lustre), I choose to build Lustre from source code. > > > I followed the steps at "Installing Lustre with a Third-Party Network > Stack" from "lustre manual 1.8". > Here is what I do. At my ~/.rpmmacros, I set my rpm working dir to > be ~/rpmbuild > > (1) Install "kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1.src.rpm". > The manual says to use "kernel-lustre-source-xxxx.rpm", but I couldn''t > find that file at Sun''s download site. > This src.rpm is installed to ~/rpmbuild/SOURCE > > (2) At ~/rpmbuild/SPECS, Run "rpmbuild -bp --target=x86_64 ./kernel-2.6.spec" > The kernel src tree is at > "~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64" > > (3) In that dir, I use the config file of currently running kernel, and then: > $ make oldconfig || make menuconfig > $ make include/asm > $ make include/linux/version.h > $ make SUBDIRS=scripts > $ make; make modules_install; make install > > This should be identical to "make rpm" and then install the > kernel-xxx.rpm and kernel-xxx.src.rpm. > > (4) Reboot into the new lustre kernel > "uname -r" gives "2.6.18-128.7.1-lustre-1.8.1-customer". > I have verified that > "/lib/modules/2.6.18-128.7.1-lustre-1.8.1-customer/build" point to > "~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64" > > > (5) Install the > "kernel-ib-devel-1.4.2-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm". > The files are installed to "/usr/src/ofa_kernel-1.4.2". > At that place, I run > $ /configure --with-core-mod --with-ipoib-mod --with-mthca-mod > --with-mlx4_core-mod --with-mlx4_inf-mod > $ make > > Then I see lots of errors like: > /usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/fs.h:9: > error: redefinition of ?inc_nlink? > include/linux/fs.h:1255: error: previous definition of ?inc_nlink? was here > In file included from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:44, > from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38, > from /usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49: > /usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/mm.h:17: > error: redefinition of ?is_vmalloc_addr? > include/linux/mm.h:332: error: previous definition of ?is_vmalloc_addr? was here > In file included from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38, > from /usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49: > /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h: In function > ?ib_dma_mapping_error?: > /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:1593: warning: > passing argument 1 of ?dma_mapping_error? makes integer from pointer > without a cast > > > I googled the error, and someone said to use OFED-1.5. > But I tried with OFED-1.4.2, OFED-1.5, and OFED-1.5.1 daily tar ball > OFED-1.5.1-20100131-0600, > and always see the same error "redefinition of ........" when I''m > compiling the kernel-ib. > > Am I wrong with the way to install kernel-ib or kernel-lustre? How > can I make lustre work with IB support? > > > (6) Build lustre succeeds without errors. > I untared "lustre-1.8.1.1.tar.gz", then > $ ./configure > --with-linux=~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64 > $ make > $ make install > > Looks both lustre kernel modules and luster userland tools/library are > installed correctly. > Is this a right way to install lustre? > > Thanks! > > > > -Neutron > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
Brian J. Murrell
2010-Feb-01 16:57 UTC
[Lustre-discuss] Unable to build kernel-ib when building lustre from source code
On Sun, 2010-01-31 at 21:12 -0500, neutron wrote:> Hi all,Hi,> (5) Install the > "kernel-ib-devel-1.4.2-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm". > The files are installed to "/usr/src/ofa_kernel-1.4.2". > At that place, I run > $ /configure --with-core-mod --with-ipoib-mod --with-mthca-mod > --with-mlx4_core-mod --with-mlx4_inf-mod > $ make > > Then I see lots of errors like: > /usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/fs.h:9: > error: redefinition of ?inc_nlink? > include/linux/fs.h:1255: error: previous definition of ?inc_nlink? was here > In file included from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:44, > from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38, > from /usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49: > /usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/mm.h:17: > error: redefinition of ?is_vmalloc_addr? > include/linux/mm.h:332: error: previous definition of ?is_vmalloc_addr? was here > In file included from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38, > from /usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49: > /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h: In function > ?ib_dma_mapping_error?: > /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:1593: warning: > passing argument 1 of ?dma_mapping_error? makes integer from pointer > without a castThese are all OFED building errors, and have nothing to do with Lustre. You *might* get help here in resolving those, but you would probably have much more luck if you go to the OFED community and ask them directly. Having said that, I do notice from the OFED 1.4.2 announcement and release notes (http://lists.openfabrics.org/pipermail/ewg/2009-August/013731.html) that the newest RHEL5 kernel supported is: - RedHat EL5 up3: 2.6.18-128.el5 Which is older than the kernel you are trying to build with: 2.6.18_128.7.1.el5> I googled the error, and someone said to use OFED-1.5.Probably because of the kernel being newer than OFED 1.4.2 supports and therefore not supported until OFED 1.5.> But I tried with OFED-1.4.2, OFED-1.5, and OFED-1.5.1 daily tar ball > OFED-1.5.1-20100131-0600, > and always see the same error "redefinition of ........" when I''m > compiling the kernel-ib.Again, this appears to be strictly an OFED issue and the OFED list might get you more help.> Am I wrong with the way to install kernel-ib or kernel-lustre? How > can I make lustre work with IB support?You are not even that far (making lustre work with IB support) yet. You are still trying to build OFED which has to be done before building Lustre. Cheers, b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100201/4c938950/attachment.bin
neutron
2010-Feb-02 16:48 UTC
[Lustre-discuss] Unable to build kernel-ib when building lustre from source code
Hi Atul, thanks a lot! Now I can build lustre, ofed. When I run mkfs.lustre, a strange thing happens: -------------- [i3:~/lustre/scripts]$sudo /usr/sbin/mkfs.lustre --fsname=lustre --reformat --mdt --mgs --device-size=100000 /dev/os/lustre Password: Permanent disk data: Target: lustre-MDTffff Index: unassigned Lustre FS: lustre Mount type: ldiskfs Flags: 0x75 (MDT MGS needs_index first_time update ) Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr Parameters: mdt.group_upcall=/usr/sbin/l_getgroups 2 6 18 formatting backing filesystem ldiskfs on /dev/os/lustre target name lustre-MDTffff 4k blocks 25000 options -i 4096 -I 512 -q -O dir_index,extents,uninit_groups -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-MDTffff -i 4096 -I 512 -q -O dir_index,extents,uninit_groups -F /dev/os/lustre 25000 sh: mke2fs: command not found mkfs.lustre FATAL: Unable to build fs /dev/os/lustre (32512) mkfs.lustre FATAL: mkfs failed 32512 --------------- It appears as if mkfs.lustre cannot find mke2fs. I have installed the e2fsprogs.rpm. "mke2fs" is in dir "/sbin", and it''s already included in the PATH. [i3:~/lustre/lustre-1.8.1]$rpm -q e2fsprogs e2fsprogs-1.41.6.sun1-0redhat [i3:~/lustre/lustre-1.8.1]$echo $PATH /sbin:/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/local/ofed/bin:/usr/local/ofed/sbin:/usr/kerberos/bin:/usr/local/bin On Sun, Jan 31, 2010 at 10:44 PM, Atul Vidwansa <Atul.Vidwansa at sun.com> wrote:> Sometime back I posted instruction on building Lustre from sources with OFED > on my blog at http://blogs.sun.com/atulvid/ . Have a look, it may be useful > to you. > > Cheers, > _Atul > > neutron wrote: >> >> Hi all, >> >> I have installed Lustre-1.8.1.1 ?from pre-built rpm packages and it >> works. Now I want to build Lustre 1.8.1.1 from source on RHEL5 >> x86_64. ?Since I want to learn Lustre( and eventually make some >> modifications in lustre), ? I choose to build Lustre from source code. >> >> >> I followed the steps at "Installing Lustre with a Third-Party Network >> Stack" from "lustre manual 1.8". >> Here is what I do. ?At my ~/.rpmmacros, ?I set my rpm working dir to >> be ~/rpmbuild >> >> (1) Install "kernel-lustre-2.6.18-128.7.1.el5_lustre.1.8.1.1.src.rpm". >> The manual says to use "kernel-lustre-source-xxxx.rpm", but I couldn''t >> find that file at Sun''s download site. >> This src.rpm is installed to ~/rpmbuild/SOURCE >> >> (2) At ~/rpmbuild/SPECS, ?Run "rpmbuild -bp --target=x86_64 >> ?./kernel-2.6.spec" >> The kernel src tree is at >> "~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64" >> >> (3) In that dir, ?I use the config file of currently running kernel, and >> then: >> ? ? ? ?$ make oldconfig || make menuconfig >> ? ? ? ?$ make include/asm >> ? ? ? ?$ make include/linux/version.h >> ? ? ? ?$ make SUBDIRS=scripts >> ? ? ? ?$ make; ?make modules_install; make install >> >> This should be identical to "make rpm" and then install the >> kernel-xxx.rpm and kernel-xxx.src.rpm. >> >> (4) Reboot into the new lustre kernel >> "uname -r" gives ?"2.6.18-128.7.1-lustre-1.8.1-customer". >> I have verified that >> "/lib/modules/2.6.18-128.7.1-lustre-1.8.1-customer/build" point to >> "~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64" >> >> >> (5) Install the >> "kernel-ib-devel-1.4.2-2.6.18_128.7.1.el5_lustre.1.8.1.1.x86_64.rpm". >> The files are installed to "/usr/src/ofa_kernel-1.4.2". >> ?At that place, I run >> ? ? ? ?$ /configure --with-core-mod --with-ipoib-mod --with-mthca-mod >> --with-mlx4_core-mod --with-mlx4_inf-mod >> ? ? ? ?$ make >> >> Then I see lots of errors like: >> >> ?/usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/fs.h:9: >> error: redefinition of ?inc_nlink? >> include/linux/fs.h:1255: error: previous definition of ?inc_nlink? was >> here >> In file included from >> /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:44, >> ? ? ? ? ? ? ? ? from /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38, >> ? ? ? ? ? ? ? ? from >> /usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49: >> >> /usr/src/ofa_kernel-1.4.2/kernel_addons/backport/2.6.18/include/linux/mm.h:17: >> error: redefinition of ?is_vmalloc_addr? >> include/linux/mm.h:332: error: previous definition of ?is_vmalloc_addr? >> was here >> In file included from >> /usr/src/ofa_kernel-1.4.2/include/rdma/ib_cache.h:38, >> ? ? ? ? ? ? ? ? from >> /usr/src/ofa_kernel-1.4.2/drivers/infiniband/core/cm.c:49: >> /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h: In function >> ?ib_dma_mapping_error?: >> /usr/src/ofa_kernel-1.4.2/include/rdma/ib_verbs.h:1593: warning: >> passing argument 1 of ?dma_mapping_error? makes integer from pointer >> without a cast >> >> >> I googled the error, and someone said to use OFED-1.5. >> But I tried with OFED-1.4.2, OFED-1.5, and OFED-1.5.1 daily tar ball >> OFED-1.5.1-20100131-0600, >> and always see the same error "redefinition of ........" ? when I''m >> compiling the kernel-ib. >> >> Am I wrong with the way to install kernel-ib or kernel-lustre? ?How >> can I make lustre work with IB support? >> >> >> (6) Build lustre succeeds without errors. >> I untared "lustre-1.8.1.1.tar.gz", then >> ? ? ? ?$ ./configure >> --with-linux=~/rpmbuild/BUILD/kernel-lustre-2.6.18/linux-2.6.18.x86_64 >> ? ? ? ?$ make >> ? ? ? ?$ make install >> >> Looks both lustre kernel modules and luster userland tools/library are >> installed correctly. >> Is this a right way to install lustre? >> >> Thanks! >> >> >> >> -Neutron >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> > >