thr3ads.net - Lustre discuss - [Lustre-discuss] Install Lustre on SLES 10 with Infiniband [Nov 2007]

If this information is useful, please help other people find it:
Share via:

Jürgen Kabelitz

2007-Nov-12 13:22 UTC

[Lustre-discuss] Install Lustre on SLES 10 with Infiniband

Hello Folks,

I need some help. We have a customer he will setup a lustre System over
infiniband. His OS are SuSE Enterprise 10 with SP1 and the infiniband software
is from OFED.
First I take the kernel source from the download site -
http://www.clusterfs.com/download.html - for sles10 :   
2.6.16-46-0.14_lustre.1.6.3smp>From the openfabrics.org I take the ofed 1.2.5.1This version is unable to compile against the kernel source from clusterfs. I
try it with the version ofed 1.0, ofed 1.2. All have the same error.

I have the questions:

Is there any system with lustre FS over Infiniband based on the Open Fabrics SW?

When there is a system, which software packets are taken?

When it is possible, give a short description how do make the software.

With regards


J. Kabelitz
sysGen GmbH
Support und Technik Clustersysteme
Am Hallacker 48
28327 Bremen
Tel.: +49 421 40966-28
Fax: +49 421 40966-33
jkabelitz at sysgen.de<mailto:jkabelitz at sysgen.de>
www.sysgen.de<http://www.sysgen.de>
Geschaeftsfuehrerin Gabriele Nikisch
Eingetragen beim Amtsgericht
Walsrode HRB 121943


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20071112/e5f4a6d0/attachment-0002.html

Liang Zhen

2007-Nov-12 14:36 UTC

head link

[Lustre-discuss] Install Lustre on SLES 10 with Infiniband

Hi,

Build Lustre against OFED 1.2.x

1. Lustre supports OFED 1.2.0 since 1.6.1

2. 1.6.1 - 1.6.3 can only support OFED 1.2.0

3. To run lustre over 1.2.5, two patches are needed:
   https://bugzilla.lustre.org/attachment.cgi?id=12508
   https://bugzilla.lustre.org/attachment.cgi?id=12629

   These patches have been landed to LNet HEAD

4. If got problem like "disagrees about version of symbol
ib_create_cq"
while
   loading ko2iblnd.ko:
   a) If <prefix>/src/openib/Module.symvers exists
      copy this file to <lustre>/lnet/klnds/o2iblnd and build lustre

   b) If <prefix>/src/openib/Module.symvers doesn''t exist
      Run this script in /lib/modules/<kernel version>/build/ and build 
lustre
      https://bugzilla.lustre.org/attachment.cgi?id=10827

   To get more details about this, please read:
   https://bugs.openfabrics.org/show_bug.cgi?id=355
   https://bugzilla.lustre.org/show_bug.cgi?id=12316

Regards
Liang

J?rgen Kabelitz wrote:>
> Hello Folks,
>
>  
>
> I need some help. We have a customer he will setup a lustre System 
> over infiniband. His OS are SuSE Enterprise 10 with SP1 and the 
> infiniband software is from OFED.
>
> First I take the kernel source from the download site - 
> http://www.clusterfs.com/download.html - for sles10 :   
>  2.6.16-46-0.14_lustre.1.6.3smp
>
> From the openfabrics.org I take the ofed 1.2.5.1
>
> This version is unable to compile against the kernel source from 
> clusterfs. I try it with the version ofed 1.0, ofed 1.2. All have the 
> same error.  
>
>  
>
> I have the questions: 
>
>  
>
> Is there any system with lustre FS over Infiniband based on the Open 
> Fabrics SW?
>
>  
>
> When there is a system, which software packets are taken?
>
>  
>
> When it is possible, give a short description how do make the software.
>
>  
>
> With regards
>
>  
>
>  
>
> J. Kabelitz
>
> sysGen GmbH
>
> Support und Techni k Clustersysteme
>
> Am Hallacker 48
>
> 28327 Bremen
>
> Tel.: +49 421 40966-28
>
> Fax: +49 421 40966-33
>
> jkabelitz at sysgen.de <mailto:jkabelitz at sysgen.de>
>
> www.sysgen.de <http://www.sysgen.de>
>
> Geschaeftsfuehrerin Gabriele Nikisch
>
> Eingetragen beim Amtsgericht
>
> Walsrode HRB 121943
>
>  
>
>  
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
>

Jürgen Kabelitz

2007-Nov-16 11:43 UTC

head link

[Lustre-discuss] Install Lustre on SLES 10 with Infiniband

when I build the OPENIB against your SLES 10 kernel, I got the following error
message:

Is this a problem from the lustre kernel or a problem from the ofed part?

  gcc
-Wp,-MD,/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/.addr.o.d 
-nostdinc -isystem /usr/lib64/gcc/x86_64-suse-linux/ 4.1.2/include -D__KERNEL__
-I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/kernel_addons/backport/2.6.16/include/ 
-I/var/tmp/OFEDRPM/BUILD/ofa_ker nel-1.2/include 
-I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/include  -Iinclude 
-Iinclude2 -I/usr/src/linux-2.6.16-46-0.14_
lustre.1.6.3/include  -include include/linux/autoconf.h  -include
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/include/linux/autoconf.h    -I/var/
tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core  -Wall -Wundef
-Wstrict-prototypes -Wno-trigraphs -Werror-implicit-function-declar ation
-fno-strict-aliasing -fno-common -ffreestanding -Os -mtune=generic -m64
-mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno- sign-compare
-fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2
-mno-3dnow -fomit-frame-pointer -fno-stack-protec tor
-Wdeclaration-after-statement -Wno-pointer-sign
-I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/include 
-I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.
2/drivers/infiniband/include 
-I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/ulp/ipoib 
-I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2
/drivers/infiniband/debug 
-I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/hw/cxgb3/core 
-I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.
2/drivers/net/cxgb3  -I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/net/rds 
-DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(a
ddr)"  -D"KBUILD_MODNAME=KBUILD_STR(ib_addr)" -c -o
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/.tmp_addr.o
/var/tmp/OFED RPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/addr.c
In file included from include2/asm/processor.h:23,
                 from
/usr/src/linux-2.6.16-46-0.14_lustre.1.6.3/include/linux/prefetch.h:14,
                 from
/usr/src/linux-2.6.16-46-0.14_lustre.1.6.3/include/linux/list.h:7,
                 from
/usr/src/linux-2.6.16-46-0.14_lustre.1.6.3/include/linux/mutex.h:13,
                 from
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/kernel_addons/backport/2.6.16/include/linux/mutex.h:5,
                 from
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/addr.c:31:
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/kernel_addons/backport/2.6.16/include/linux/cpumask.h:6:1:
warning: "for_each_possible_cpu" redefined In file included from
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/kernel_addons/backport/2.6.16/include/linux/cpumask.h:4,
                 from include2/asm/processor.h:23,
                 from
/usr/src/linux-2.6.16-46-0.14_lustre.1.6.3/include/linux/prefetch.h:14,
                 from
/usr/src/linux-2.6.16-46-0.14_lustre.1.6.3/include/linux/list.h:7,
                 from
/usr/src/linux-2.6.16-46-0.14_lustre.1.6.3/include/linux/mutex.h:13,
                 from
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/kernel_addons/backport/2.6.16/include/linux/mutex.h:5,
                 from
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/addr.c:31:
/usr/src/linux-2.6.16-46-0.14_lustre.1.6.3/include/linux/cpumask.h:411:1:
warning: this is the location of the previous definition In file included from
/usr/src/linux-2.6.16-46-0.14_lustre.1.6.3/include/linux/inetdevice.h:7,
                 from
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/kernel_addons/backport/2.6.16/include/linux/inetdevice.h:4,
                 from
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/addr.c:32:
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/kernel_addons/backport/2.6.16/include/linux/netdevice.h:7:
error: redefinition of ?netif_tx_lock?
/usr/src/linux-2.6.16-46-0.14_lustre.1.6.3/include/linux/netdevice.h:927: error:
previous definition of ?netif_tx_lock? was here
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/kernel_addons/backport/2.6.16/include/linux/netdevice.h:
In function ?netif_tx_lock?:
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/kernel_addons/backport/2.6.16/include/linux/netdevice.h:8:
error: ?struct net_device? has no member n amed ?xmit_lock?
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/kernel_addons/backport/2.6.16/include/linux/netdevice.h:
At top level:
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/kernel_addons/backport/2.6.16/include/linux/netdevice.h:13:
error: redefinition of ?netif_tx_unlock?
/usr/src/linux-2.6.16-46-0.14_lustre.1.6.3/include/linux/netdevice.h:947: error:
previous definition of ?netif_tx_unlock? was here
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/kernel_addons/backport/2.6.16/include/linux/netdevice.h:
In function ?netif_tx_unlock?:
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/kernel_addons/backport/2.6.16/include/linux/netdevice.h:15:
error: ?struct net_device? has no member named ?xmit_lock?
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/addr.c: At top
level:
/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/addr.c:61:
warning: initialization from incompatible pointer type
make[6]: ***
[/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/addr.o] Error 1
make[5]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core]
Error 2
make[4]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband] Error 2
make[3]: *** [_module_/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2] Error 2
make[2]: *** [modules] Error 2
make[1]: *** [modules] Error 2
make[1]: Leaving directory
`/usr/src/linux-2.6.16-46-0.14_lustre.1.6.3-obj/x86_64/smp''
make: *** [kernel] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.67752 (%install)


RPM build errors:
    user vlad does not exist - using root
    group vlad does not exist - using root
    user vlad does not exist - using root
    group vlad does not exist - using root
    Bad exit status from /var/tmp/rpm-tmp.67752 (%install)
ERROR: Failed executing "rpmbuild --rebuild --define ''_topdir
/var/tmp/OFEDRPM'' --define ''_prefix /usr'' --define
''build_root /var/tmp/OFED''
 --define ''configure_options --with-cxgb3-mod --with-ipoib-mod
--with-mthca-mod --with-sdp-mod --with-srp-mod --with-core-mod --with-user_m
ad-mod --with-user_access-mod --with-addr_trans-mod --with-rds-mod ''
--define ''KVERSION 2.6.16-46-0.14_lustre.1.6.3smp'' --define
''KSRC /lib
/modules/2.6.16-46-0.14_lustre.1.6.3smp/build'' --define
''build_kernel_ib 1'' --define ''build_kernel_ib_devel
1'' --define ''NETWORK_CONF_DIR /
etc/sysconfig/network'' --define ''modprobe_update 1''
--define ''include_ipoib_conf 1''
/usr/src/OFED-1.2/SRPMS/ofa_kernel-1.2-0.src.rpm"
~



J. Kabelitz
sysGen GmbH
Support und Technik Clustersysteme
Am Hallacker 48
28327 Bremen
Tel.: +49 421 40966-28
Fax: +49 421 40966-33
jkabelitz at sysgen.de<mailto:jkabelitz at sysgen.de>
www.sysgen.de<http://www.sysgen.de>
Geschaeftsfuehrerin Gabriele Nikisch
Eingetragen beim Amtsgericht
Walsrode HRB 121943


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20071116/fa215591/attachment-0002.html

Lustre discuss - Nov 2007 - Install Lustre on SLES 10 with Infiniband

[Lustre-discuss] Install Lustre on SLES 10 with Infiniband

[Lustre-discuss] Install Lustre on SLES 10 with Infiniband

[Lustre-discuss] Install Lustre on SLES 10 with Infiniband