thr3ads.net - Lustre discuss - [Lustre-discuss] 1.6b5 with InfiniBand [Dec 2006]

If this information is useful, please help other people find it:
Share via:

Mirko Benz

2006-Dec-19 06:15 UTC

[Lustre-discuss] 1.6b5 with InfiniBand

Hi,

We are testing 1.6b5 for a InfiniBand cluster with RHEL 4. We use the 
binaries provides by CFS and use OFED 1.1 as the IB stack.

At several times some of the clients hang during fs mount or when an OST 
is added (see log).
Error:
LustreError: 1776:0:(o2iblnd_cb.c:2314:kiblnd_rejected()) 10.0.90.8@o2ib 
rejected: reason 8, size 148

from OFED:
enum ib_cm_rej_reason {
       IB_CM_REJ_INVALID_SERVICE_ID            = 8,

Once an IPoIB ping is started to the corresponding OST the client 
continues. Afterwards it is quite stable.

Any idea how this could be fixed?

Thanks,
Mirko

-------------- next part --------------
Lustre:   mount data:
Lustre: profile: testfs-client
Lustre: device:  10.0.90.10@o2ib:/testfs
Lustre: flags:   2
Lustre:   0 UP mgc MGC10.0.90.10@o2ib 438411f9-d2cc-f576-9a5d-bc927badfa60 5
Lustre:   1 UP lov testfs-clilov-0000010075688000
7255c262-21e0-f804-91dd-2e8008cc166a 3
Lustre:   2 UP mdc testfs-MDT0000-mdc-0000010075688000
7255c262-21e0-f804-91dd-2e8008cc166a 4
Lustre:   3 UP osc testfs-OST0000-osc-0000010075688000
7255c262-21e0-f804-91dd-2e8008cc166a 4
Lustre:   4 UP osc testfs-OST0001-osc-0000010075688000
7255c262-21e0-f804-91dd-2e8008cc166a 4
Lustre: mount 10.0.90.10@o2ib:/testfs complete
Lustre: client 0000010075688000 umount complete
Lustre:   mount data:
Lustre: profile: testfs-client
Lustre: device:  10.0.90.10@o2ib:/testfs
Lustre: flags:   2
Lustre:   0 UP mgc MGC10.0.90.10@o2ib caf868ce-f8dc-8c83-ecd4-caf4a75378f2 5
Lustre:   1 UP lov testfs-clilov-000001007eaba800
3119a81c-5954-8b92-edab-5e38c9f7743d 3
Lustre:   2 UP mdc testfs-MDT0000-mdc-000001007eaba800
3119a81c-5954-8b92-edab-5e38c9f7743d 4
Lustre:   3 UP osc testfs-OST0000-osc-000001007eaba800
3119a81c-5954-8b92-edab-5e38c9f7743d 4
Lustre:   4 UP osc testfs-OST0001-osc-000001007eaba800
3119a81c-5954-8b92-edab-5e38c9f7743d 4
Lustre:   5 UP osc testfs-OST0002-osc-000001007eaba800
3119a81c-5954-8b92-edab-5e38c9f7743d 4
Lustre:   6 UP osc testfs-OST0003-osc-000001007eaba800
3119a81c-5954-8b92-edab-5e38c9f7743d 4
Lustre:   7 UP osc testfs-OST0004-osc-000001007eaba800
3119a81c-5954-8b92-edab-5e38c9f7743d 4
Lustre:   8 UP osc testfs-OST0005-osc-000001007eaba800
3119a81c-5954-8b92-edab-5e38c9f7743d 4
Lustre:   9 UP osc testfs-OST0006-osc-000001007eaba800
3119a81c-5954-8b92-edab-5e38c9f7743d 4
Lustre:  10 UP osc testfs-OST0007-osc-000001007eaba800
3119a81c-5954-8b92-edab-5e38c9f7743d 4
Lustre: mount 10.0.90.10@o2ib:/testfs complete
LustreError: 1776:0:(o2iblnd_cb.c:2314:kiblnd_rejected()) 10.0.90.8@o2ib
rejected: reason 8, size 148
LustreError: 1776:0:(o2iblnd_cb.c:1935:kiblnd_peer_connect_failed()) Deleting
messages for 10.0.90.8@o2ib: connection failed
LustreError: 1776:0:(events.c:51:request_out_callback()) @@@ type 4, status -113
LustreError: 1776:0:(events.c:51:request_out_callback()) Skipped 1 previous
similar message
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent
at 1166521780, 0s ago)
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) Skipped 1
previous similar message
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
LustreError: 5171:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Rx from
10.0.90.8@o2ib failed: 5
LustreError: 1776:0:(o2iblnd_cb.c:2314:kiblnd_rejected()) 10.0.90.8@o2ib
rejected: reason 8, size 148
LustreError: 1776:0:(o2iblnd_cb.c:1935:kiblnd_peer_connect_failed()) Deleting
messages for 10.0.90.8@o2ib: connection failed
LustreError: 1776:0:(events.c:51:request_out_callback()) @@@ type 4, status -113
LustreError: 1776:0:(events.c:51:request_out_callback()) Skipped 3 previous
similar messages
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent
at 1166521805, 0s ago)
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) Skipped 3
previous similar messages
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
LustreError: 5171:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Rx from
10.0.90.8@o2ib failed: 5
LustreError: 5171:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Skipped 5 previous
similar messages
LustreError: 1775:0:(o2iblnd_cb.c:2314:kiblnd_rejected()) 10.0.90.8@o2ib
rejected: reason 8, size 148
LustreError: 1775:0:(o2iblnd_cb.c:1935:kiblnd_peer_connect_failed()) Deleting
messages for 10.0.90.8@o2ib: connection failed
LustreError: 1775:0:(events.c:51:request_out_callback()) @@@ type 4, status -113
LustreError: 1775:0:(events.c:51:request_out_callback()) Skipped 3 previous
similar messages
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent
at 1166521830, 0s ago)
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) Skipped 3
previous similar messages
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
LustreError: 5170:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Rx from
10.0.90.8@o2ib failed: 5
LustreError: 5170:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Skipped 1 previous
similar message
LustreError: 1775:0:(o2iblnd_cb.c:2314:kiblnd_rejected()) 10.0.90.8@o2ib
rejected: reason 8, size 148
LustreError: 1775:0:(o2iblnd_cb.c:1935:kiblnd_peer_connect_failed()) Deleting
messages for 10.0.90.8@o2ib: connection failed
LustreError: 1775:0:(events.c:51:request_out_callback()) @@@ type 4, status -113
LustreError: 1775:0:(events.c:51:request_out_callback()) Skipped 3 previous
similar messages
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent
at 1166521855, 0s ago)
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) Skipped 3
previous similar messages
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
LustreError: 5171:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Rx from
10.0.90.8@o2ib failed: 5
LustreError: 5171:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Skipped 15 previous
similar messages
LustreError: 1776:0:(o2iblnd_cb.c:2314:kiblnd_rejected()) 10.0.90.8@o2ib
rejected: reason 8, size 148
LustreError: 1776:0:(o2iblnd_cb.c:1935:kiblnd_peer_connect_failed()) Deleting
messages for 10.0.90.8@o2ib: connection failed
LustreError: 1776:0:(events.c:51:request_out_callback()) @@@ type 4, status -113
LustreError: 1776:0:(events.c:51:request_out_callback()) Skipped 3 previous
similar messages
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent
at 1166521880, 0s ago)
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) Skipped 3
previous similar messages
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
LustreError: 5171:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Rx from
10.0.90.8@o2ib failed: 5
LustreError: 5171:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Skipped 15 previous
similar messages
LustreError: 1776:0:(o2iblnd_cb.c:2314:kiblnd_rejected()) 10.0.90.8@o2ib
rejected: reason 8, size 148
LustreError: 1776:0:(o2iblnd_cb.c:1935:kiblnd_peer_connect_failed()) Deleting
messages for 10.0.90.8@o2ib: connection failed
LustreError: 1776:0:(events.c:51:request_out_callback()) @@@ type 4, status -113
LustreError: 1776:0:(events.c:51:request_out_callback()) Skipped 3 previous
similar messages
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent
at 1166521905, 0s ago)
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) Skipped 3
previous similar messages
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
LustreError: 5171:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Rx from
10.0.90.8@o2ib failed: 5
LustreError: 5171:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Skipped 17 previous
similar messages
LustreError: 1775:0:(o2iblnd_cb.c:2314:kiblnd_rejected()) 10.0.90.8@o2ib
rejected: reason 8, size 148
LustreError: 1775:0:(o2iblnd_cb.c:1935:kiblnd_peer_connect_failed()) Deleting
messages for 10.0.90.8@o2ib: connection failed
LustreError: 1775:0:(events.c:51:request_out_callback()) @@@ type 4, status -113
LustreError: 1775:0:(events.c:51:request_out_callback()) Skipped 3 previous
similar messages
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent
at 1166521930, 0s ago)
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) Skipped 3
previous similar messages
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
LustreError: 5170:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Rx from
10.0.90.8@o2ib failed: 5
LustreError: 5170:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Skipped 1 previous
similar message
LustreError: 1775:0:(o2iblnd_cb.c:2314:kiblnd_rejected()) 10.0.90.8@o2ib
rejected: reason 8, size 148
LustreError: 1775:0:(o2iblnd_cb.c:1935:kiblnd_peer_connect_failed()) Deleting
messages for 10.0.90.8@o2ib: connection failed
LustreError: 1775:0:(events.c:51:request_out_callback()) @@@ type 4, status -113
LustreError: 1775:0:(events.c:51:request_out_callback()) Skipped 3 previous
similar messages
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent
at 1166521955, 0s ago)
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) Skipped 3
previous similar messages
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
LustreError: 5171:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Rx from
10.0.90.8@o2ib failed: 5
LustreError: 5171:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Skipped 15 previous
similar messages
LustreError: 1776:0:(o2iblnd_cb.c:2314:kiblnd_rejected()) 10.0.90.8@o2ib
rejected: reason 8, size 148
LustreError: 1776:0:(o2iblnd_cb.c:1935:kiblnd_peer_connect_failed()) Deleting
messages for 10.0.90.8@o2ib: connection failed
LustreError: 1776:0:(events.c:51:request_out_callback()) @@@ type 4, status -113
LustreError: 1776:0:(events.c:51:request_out_callback()) Skipped 3 previous
similar messages
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent
at 1166521980, 0s ago)
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) Skipped 3
previous similar messages
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
LustreError: 5171:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Rx from
10.0.90.8@o2ib failed: 5
LustreError: 5171:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Skipped 15 previous
similar messages
LustreError: 1776:0:(o2iblnd_cb.c:2314:kiblnd_rejected()) 10.0.90.8@o2ib
rejected: reason 8, size 148
LustreError: 1776:0:(o2iblnd_cb.c:1935:kiblnd_peer_connect_failed()) Deleting
messages for 10.0.90.8@o2ib: connection failed
LustreError: 1776:0:(events.c:51:request_out_callback()) @@@ type 4, status -113
LustreError: 1776:0:(events.c:51:request_out_callback()) Skipped 3 previous
similar messages
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent
at 1166522005, 0s ago)
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) Skipped 3
previous similar messages
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
LustreError: 5170:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Rx from
10.0.90.8@o2ib failed: 5
LustreError: 5170:0:(o2iblnd_cb.c:455:kiblnd_rx_complete()) Skipped 15 previous
similar messages
LustreError: 1775:0:(o2iblnd_cb.c:2314:kiblnd_rejected()) 10.0.90.8@o2ib
rejected: reason 8, size 148
LustreError: 1775:0:(o2iblnd_cb.c:1935:kiblnd_peer_connect_failed()) Deleting
messages for 10.0.90.8@o2ib: connection failed
LustreError: 1775:0:(events.c:51:request_out_callback()) @@@ type 4, status -113
LustreError: 1775:0:(events.c:51:request_out_callback()) Skipped 3 previous
similar messages
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent
at 1166522030, 0s ago)
LustreError: 5909:0:(client.c:950:ptlrpc_expire_one_request()) Skipped 3
previous similar messages
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0
Lustre: 5909:0:(peer.c:238:lnet_debug_peer()) 10.0.90.8@o2ib              2   
up     8     8     8     8     6 0

Mirko Benz

2006-Dec-21 08:46 UTC

head link

[Lustre-discuss] 1.6b5 with InfiniBand

Hi,

Please look at the attached log to see what I mean. The behavior and 
dmesg output is similar when using your sequence of commands.

We have removed the two Mellanox cards from the OSS and put a single 
Voltaire card in it.
This seems to work. I could connect 60 nodes without errors. We will 
further investigate this to better understand the cause of the original 
problem.

Thanks,
Mirko

Eric Barton schrieb:>> At several times some of the clients hang during fs mount or 
>> when an OST 
>> is added (see log).
>> Error:
>> LustreError: 1776:0:(o2iblnd_cb.c:2314:kiblnd_rejected()) 
>> 10.0.90.8@o2ib 
>> rejected: reason 8, size 148
>>
>> from OFED:
>> enum ib_cm_rej_reason {
>>        IB_CM_REJ_INVALID_SERVICE_ID            = 8,
>>
>> Once an IPoIB ping is started to the corresponding OST the client 
>> continues. Afterwards it is quite stable.
>>
>> Any idea how this could be fixed?
>>     
>
> Are you sure the OST is up and running before you attempt to mount it?
>
> Do you get the same error if you do the following on the client before you
> try to mount...
>
> 1. modprobe lnet
> 2. lctl net up
> 3. lctl ping 10.0.90.8@02ib
>
>
>                 Cheers,
>                         Eric
>
>   
-------------- next part --------------
[root@mgt xiranet]# psh node107-node110 mount -t lustre 10.0.90.9@o2ib0:/testfs
/mnt/testfs
node107:        mount.lustre: mount 10.0.90.9@o2ib:/testfs at /mnt/testfs
failed: Input/output error
node107:        Is the MGS running?
node109:        mount.lustre: mount 10.0.90.9@o2ib:/testfs at /mnt/testfs
failed: Input/output error
node109:        Is the MGS running?
[root@mgt xiranet]# psh node107-node110 mount | grep testfs
node110:        10.0.90.9@o2ib:/testfs on /mnt/testfs type lustre (rw)
node108:        10.0.90.9@o2ib:/testfs on /mnt/testfs type lustre (rw)
[root@mgt xiranet]# psh node107-node110 ping -c 3 10.0.90.9 | grep loss
node110:        3 packets transmitted, 3 received, 0% packet loss, time 1999ms
node107:        3 packets transmitted, 3 received, 0% packet loss, time 1999ms
node109:        3 packets transmitted, 3 received, 0% packet loss, time 1999ms
node108:        3 packets transmitted, 3 received, 0% packet loss, time 1999ms
[root@mgt xiranet]# psh node107-node110 mount -t lustre 10.0.90.9@o2ib0:/testfs
/mnt/testfs
node108:        mount.lustre: according to /etc/mtab 10.0.90.9@o2ib:/testfs is
already mounted on /mnt/testfs
node110:        mount.lustre: according to /etc/mtab 10.0.90.9@o2ib:/testfs is
already mounted on /mnt/testfs
[root@mgt xiranet]# psh node107-node110 mount -t lustre 10.0.90.9@o2ib0:/testfs
/mnt/testfs
node110:        mount.lustre: according to /etc/mtab 10.0.90.9@o2ib:/testfs is
already mounted on /mnt/testfs
node107:        mount.lustre: according to /etc/mtab 10.0.90.9@o2ib:/testfs is
already mounted on /mnt/testfs
node108:        mount.lustre: according to /etc/mtab 10.0.90.9@o2ib:/testfs is
already mounted on /mnt/testfs
node109:        mount.lustre: according to /etc/mtab 10.0.90.9@o2ib:/testfs is
already mounted on /mnt/testfs
[root@mgt xiranet]#

Lustre discuss - Dec 2006 - 1.6b5 with InfiniBand

[Lustre-discuss] 1.6b5 with InfiniBand

[Lustre-discuss] 1.6b5 with InfiniBand