Lustre discuss - Jun 2006 - kopeniblnd.ko module missing in lustre-1.4.6.2

Hi All,
       we have installed lustre-1.4.6.2 on RHELAS4-U2,
facing some issue with "lctl network up".

  it gives error saying network down, if make tcp0
as protocol in modules.conf, then it can be up.

 searching the docs, found "kopeniblnd.ko" missing
in my installation, is it replaced with some
other module??????

pls clarify,

thanks in advance,
-bala-



__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com

sorry to reply my own post, I also found the following
error message in "dmesg", running RHELAS4-U2 on Intel
Xeon EM64T

[root@oss01 ~]# modprobe lnet
[root@oss01 ~]# lctl network up

[root@oss01 ~]# dmesg |grep lnd
LustreError:
4426:0:(api-ni.c:985:lnet_startup_lndnis()) Can''t load
LND openib, module kopeniblnd, rc=256

hope someone can give me some suggestion/help

-bala-

--- Bala <cppbala@yahoo.com> wrote:
> Hi All,
>        we have installed lustre-1.4.6.2 on
> RHELAS4-U2,
> facing some issue with "lctl network up".
> 
>   it gives error saying network down, if make tcp0
> as protocol in modules.conf, then it can be up.
> 
>  searching the docs, found "kopeniblnd.ko" missing
> in my installation, is it replaced with some
> other module??????
> 
> pls clarify,
> 
> thanks in advance,
> -bala-
> 
> 
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam
> protection around 
> http://mail.yahoo.com 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss@clusterfs.com
>
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
> 

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com

Hi,
   Eric thanks for pointing me, as you mentioned
I have tried compiling with gen1 headers

I was getting the following error

/scratch1/lustre/src/lustre-1.4.6.2/lnet/klnds/openiblnd/openiblnd_cb.c:
In function `kibnal_post_rx'':
/scratch1/lustre/src/lustre-1.4.6.2/lnet/klnds/openiblnd/openiblnd_cb.c:227:
error: too many arguments to function `ib_receive''
/scratch1/lustre/src/lustre-1.4.6.2/lnet/klnds/openiblnd/openiblnd_cb.c:
In function `kibnal_check_sends'':
/scratch1/lustre/src/lustre-1.4.6.2/lnet/klnds/openiblnd/openiblnd_cb.c:697:
error: too many arguments to function `ib_send''
make[7]: *** `/scratch1/lustre/src/lustre-1.4.6.2''
make: *** [all] Error 2

then I have chnaged ib_receive, ib_send in
openiblnd_cb.c file 

from  rc = ib_receive (conn->ibc_qp, &rx->rx_sp, 1,
NULL);  
to  rc = ib_receive (conn->ibc_qp, &rx->rx_sp, 1);

from  rc = ib_send (conn->ibc_qp, &tx->tx_sp[i], 1,
NULL);
to rc = ib_send (conn->ibc_qp, &tx->tx_sp[i], 1);

then it compiles and installs

but while loading the module

  #modprobe -v lnet  (loads ok with libcfs)
  ## modprobe -v kopeniblnd
insmod
/lib/modules/2.6.9-34.EL_lustre.1.4.6.2smp/kernel/net/lustre/kopeniblnd.ko
FATAL: Error inserting kopeniblnd
(/lib/modules/2.6.9-34.EL_lustre.1.4.6.2smp/kernel/net/lustre/kopeniblnd.ko):
Unknown symbol in module, or unknown parameter (see
dmesg)

the dmesg output is
# dmesg |grep lnd
LustreError:
4426:0:(api-ni.c:985:lnet_startup_lndnis()) Can''t load
LND openib, module kopeniblnd, rc=256

kopeniblnd: Unknown symbol ib_cached_gid_get
kopeniblnd: Unknown symbol tsKernelTrace
kopeniblnd: Unknown symbol ib_pd_destroy
kopeniblnd: Unknown symbol ib_qp_destroy
kopeniblnd: Unknown symbol ib_fmr_pool_destroy
LustreError:
3902:0:(api-ni.c:985:lnet_startup_lndnis()) Can''t load
LND
openib, module kopeniblnd, rc=256

can anyone help me to solve this issue.

thanks in advance,
-bala-



--- Eric Barton <eeb@bartonsoftware.com> wrote:
> Have you built and installed the openib LND
> (./configure --with
> openib=<path to openib gen1 / mellanox gold
> headers>)?
> 
> 
> On Tue, 2006-06-13 at 06:53 -0700, Bala wrote:
> > sorry to reply my own post, I also found the
> following
> > error message in "dmesg", running RHELAS4-U2 on
> Intel
> > Xeon EM64T
> > 
> > [root@oss01 ~]# modprobe lnet
> > [root@oss01 ~]# lctl network up
> > 
> > [root@oss01 ~]# dmesg |grep lnd
> > LustreError:
> > 4426:0:(api-ni.c:985:lnet_startup_lndnis()) Can''t
> load
> > LND openib, module kopeniblnd, rc=256
> > 
> > hope someone can give me some suggestion/help
> > 
> > -bala-
> > 
> > --- Bala <cppbala@yahoo.com> wrote:
> > 
> > > Hi All,
> > >        we have installed lustre-1.4.6.2 on
> > > RHELAS4-U2,
> > > facing some issue with "lctl network up".
> > > 
> > >   it gives error saying network down, if make
> tcp0
> > > as protocol in modules.conf, then it can be up.
> > > 
> > >  searching the docs, found "kopeniblnd.ko"
> missing
> > > in my installation, is it replaced with some
> > > other module??????
> > > 
> > > pls clarify,
> > > 
> > > thanks in advance,
> > > -bala-
> > > 
> > > 
> > > 
> > >
> __________________________________________________
> > > Do You Yahoo!?
> > > Tired of spam?  Yahoo! Mail has the best spam
> > > protection around 
> > > http://mail.yahoo.com 
> > > _______________________________________________
> > > Lustre-discuss mailing list
> > > Lustre-discuss@clusterfs.com
> > >
> >
>
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
> > > 
> > 
> > 
> > __________________________________________________
> > Do You Yahoo!?
> > Tired of spam?  Yahoo! Mail has the best spam
> protection around 
> > http://mail.yahoo.com 
> > _______________________________________________
> > Lustre-discuss mailing list
> > Lustre-discuss@clusterfs.com
> >
>
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
> 
> 
> 

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam
protection around 
http://mail.yahoo.com 

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com

Hi,

--- Eric Barton <eeb@bartonsoftware.com> wrote:
> Have you built and installed the openib LND
> (./configure --with
> openib=<path to openib gen1 / mellanox gold
> headers>)?
  as you have mentioned compiled with openiblnd
with gen1 headers, since the lustre kernel not 
aware of the header files we are getting 
"Unknown symbol" error, is there anyway we can
avoid this error and load the kopeniblnd module??

the error message is
---------------------------
#modprobe -v lnet  (loads ok with libcfs)
  ## modprobe -v kopeniblnd
insmod
/lib/modules/2.6.9-34.EL_lustre.1.4.6.2smp/kernel/net/lustre/kopeniblnd.ko
FATAL: Error inserting kopeniblnd
(/lib/modules/2.6.9-34.EL_lustre.1.4.6.2smp/kernel/net/lustre/kopeniblnd.ko):
Unknown symbol in module, or unknown parameter (see
dmesg)

the dmesg output is
# dmesg |grep lnd
LustreError:
4426:0:(api-ni.c:985:lnet_startup_lndnis()) Can''t load
LND openib, module kopeniblnd, rc=256

kopeniblnd: Unknown symbol ib_cached_gid_get
kopeniblnd: Unknown symbol tsKernelTrace
kopeniblnd: Unknown symbol ib_pd_destroy
kopeniblnd: Unknown symbol ib_qp_destroy
kopeniblnd: Unknown symbol ib_fmr_pool_destroy
LustreError:
3902:0:(api-ni.c:985:lnet_startup_lndnis()) Can''t load
LND
openib, module kopeniblnd, rc=256
---------------------------

thanks,
-bala-

> 
> 
> On Tue, 2006-06-13 at 06:53 -0700, Bala wrote:
> > sorry to reply my own post, I also found the
> following
> > error message in "dmesg", running RHELAS4-U2 on
> Intel
> > Xeon EM64T
> > 
> > [root@oss01 ~]# modprobe lnet
> > [root@oss01 ~]# lctl network up
> > 
> > [root@oss01 ~]# dmesg |grep lnd
> > LustreError:
> > 4426:0:(api-ni.c:985:lnet_startup_lndnis()) Can''t
> load
> > LND openib, module kopeniblnd, rc=256
> > 
> > hope someone can give me some suggestion/help
> > 
> > -bala-
> > 
> > --- Bala <cppbala@yahoo.com> wrote:
> > 
> > > Hi All,
> > >        we have installed lustre-1.4.6.2 on
> > > RHELAS4-U2,
> > > facing some issue with "lctl network up".
> > > 
> > >   it gives error saying network down, if make
> tcp0
> > > as protocol in modules.conf, then it can be up.
> > > 
> > >  searching the docs, found "kopeniblnd.ko"
> missing
> > > in my installation, is it replaced with some
> > > other module??????
> > > 
> > > pls clarify,
> > > 
> > > thanks in advance,
> > > -bala-
> > > 
> > > 
> > > 
> > >
> __________________________________________________
> > > Do You Yahoo!?
> > > Tired of spam?  Yahoo! Mail has the best spam
> > > protection around 
> > > http://mail.yahoo.com 
> > > _______________________________________________
> > > Lustre-discuss mailing list
> > > Lustre-discuss@clusterfs.com
> > >
> >
>
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
> > > 
> > 
> > 
> > __________________________________________________
> > Do You Yahoo!?
> > Tired of spam?  Yahoo! Mail has the best spam
> protection around 
> > http://mail.yahoo.com 
> > _______________________________________________
> > Lustre-discuss mailing list
> > Lustre-discuss@clusterfs.com
> >
>
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
> 
> 
> 

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com

On Thu, 2006-06-15 at 06:25 -0700, Bala wrote:
> kopeniblnd: Unknown symbol ib_cached_gid_get
> kopeniblnd: Unknown symbol tsKernelTrace
> kopeniblnd: Unknown symbol ib_pd_destroy
> kopeniblnd: Unknown symbol ib_qp_destroy
> kopeniblnd: Unknown symbol ib_fmr_pool_destroy
These symbols are defined by the OpenIB gen1 / Mellanox Gold kernel
modules...

--- Eric Barton <eeb@bartonsoftware.com> wrote:
> On Thu, 2006-06-15 at 06:25 -0700, Bala wrote:
> 
> > kopeniblnd: Unknown symbol ib_cached_gid_get
> > kopeniblnd: Unknown symbol tsKernelTrace
> > kopeniblnd: Unknown symbol ib_pd_destroy
> > kopeniblnd: Unknown symbol ib_qp_destroy
> > kopeniblnd: Unknown symbol ib_fmr_pool_destroy
> 
> These symbols are defined by the OpenIB gen1 /
> Mellanox Gold kernel
> modules...
   any suggestion to solve this issue???, this comes
only if I try to build/insert kopeniblnd module, got
the header files from clusterfs site, am I using the 
right header files??, need your help to get ride
this error.

thanks in advance,
-bala-


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com

This problem may also occur if the kernel that is running does not match 
against the kernel source with which the kernel-module was built 
against. Safe is to build the kernel from the source, install, and 
reboot with the kernel built. Then build the kernel-modules, and try 
loading them. 
 
Also, when you build Lustre against openib, 
make  /usr/src/linux/drivers/infiniband to point Gen1 source. 
 
kimi:/usr/src/linux/drivers # ls -l infiniband 
lrwxrwxrwx  1 root root 17 Sep  8  2005 infiniband -> /usr/src/ib-1.8.0 
 
HTH, 
-Kums
>>>Bala <cppbala@yahoo.com> 06/15/06 8:54 pm >>>
--- Eric Barton <eeb@bartonsoftware.com> wrote:
>On Thu, 2006-06-15 at 06:25 -0700, Bala wrote:
>
>>kopeniblnd: Unknown symbol ib_cached_gid_get
>>kopeniblnd: Unknown symbol tsKernelTrace
>>kopeniblnd: Unknown symbol ib_pd_destroy
>>kopeniblnd: Unknown symbol ib_qp_destroy
>>kopeniblnd: Unknown symbol ib_fmr_pool_destroy
>
>These symbols are defined by the OpenIB gen1 /
>Mellanox Gold kernel
>modules...
  any suggestion to solve this issue???, this comes
only if I try to build/insert kopeniblnd module, got
the header files from clusterfs site, am I using the
right header files??, need your help to get ride
this error.

thanks in advance,
-bala-



Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

Lustre-discuss mailing list
Lustre-discuss@clusterfs.com
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20060616/316d4c28/attachment-0001.html

Hi Bala,

Apparently your kernel is configured with module version support.
There''re two ways to solve this issue:
 - rebuild your kernel without module version support: set 
CONFIG_MODVERSIONS=n in .config;
- when trying to load kopeniblnd, use ''modprobe --force-modversion 
kopeniblnd'' instead.

Bala wrote:
> --- Eric Barton <eeb@bartonsoftware.com> wrote:
>   
>> On Thu, 2006-06-15 at 06:25 -0700, Bala wrote:
>>
>>     
>>> kopeniblnd: Unknown symbol ib_cached_gid_get
>>> kopeniblnd: Unknown symbol tsKernelTrace
>>> kopeniblnd: Unknown symbol ib_pd_destroy
>>> kopeniblnd: Unknown symbol ib_qp_destroy
>>> kopeniblnd: Unknown symbol ib_fmr_pool_destroy
>>>       
>> These symbols are defined by the OpenIB gen1 /
>> Mellanox Gold kernel
>> modules...
>>     
>
>    any suggestion to solve this issue???, this comes
> only if I try to build/insert kopeniblnd module, got
> the header files from clusterfs site, am I using the 
> right header files??, need your help to get ride
> this error.
>
> thanks in advance,
> -bala-
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around 
> http://mail.yahoo.com 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss@clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
>
>   

-- 
Cheers,
     Wang Yibin

Wang Yibin wrote:
> Hi Bala,
>
> Apparently your kernel is configured with module version support.
> There''re two ways to solve this issue:
> - rebuild your kernel without module version support: set 
> CONFIG_MODVERSIONS=n in .config;
Ah, it should be replace ''CONFIG_MODVERSIONS=y'' with
''#
CONFIG_MODVERSIONS is not set''
Sorry about my programmer nature...
> - when trying to load kopeniblnd, use ''modprobe --force-modversion
> kopeniblnd'' instead.
>
-- 
Cheers,
     Wang Yibin