thr3ads.net - Lustre discuss - [Lustre-discuss] Multihoned Problem, can mount o2ib but not tcp [Oct 2009]

If this information is useful, please help other people find it:
Share via:

Mike Hanby

2009-Oct-29 22:13 UTC

[Lustre-discuss] Multihoned Problem, can mount o2ib but not tcp

Howdy,

I have a working Lustre file system set up using Infiniband:
 1 x MDS/MGS server
 2 x OSS/OST servers, in active active failover
 25 x client nodes

All of these systems use Infiniband with Lustre.

Now, I have 60 older compute nodes that I''d like to add to the system.
These only have Gigabit Ethernet.

I''ve added the tcp network to lnet (see steps below), but when I
attempt to mount my luster filesystem on the tcp client it fails, and the error
in /var/log/messages seems to indicate that it''s trying to use o2ib,
and not tcp:

# mount -t lustre 172.20.20.30 at tcp:/lustre /lustre
mount.lustre: mount 172.20.20.30 at tcp:/lustre at /lustre failed: No such file
or directory
Is the MGS specification correct?
Is the filesystem name correct?
If upgrading, is the copied client log valid? (see upgrade docs)

kernel: LustreError: 2860:0:(events.c:460:ptlrpc_uuid_to_peer()) No NID found
for 172.20.21.30 at o2ib
kernel: LustreError: 2860:0:(client.c:69:ptlrpc_uuid_to_connection()) cannot
find peer 172.20.21.30 at o2ib!
kernel: LustreError: 2860:0:(ldlm_lib.c:329:client_obd_setup()) can''t
add initial connection
kernel: LustreError: 2860:0:(obd_config.c:370:class_setup()) setup
lustre-MDT0000-mdc-ffff81007eb82400 failed (-2)
kernel: LustreError: 2860:0:(obd_config.c:1197:class_config_llog_handler()) Err
-2 on cfg command:
kernel: LustreError: 15c-8: MGC172.20.20.30 at tcp: The configuration from log
''lustre-client'' failed (-2). This may be the result of
communication errors between this node and the MGS, a bad configuration, or
other errors. See the syslog for more information.

Here''s my modprobe.conf entry for the OSS/MDS servers (is order
important here or tcp and o2ib?)
options lnet networks="tcp0(eth0),o2ib(ib0)"
options ko2iblnd concurrent_sends=7
options ptlrpc at_max=600 
options ost oss_num_threads=512

modprobe.conf file for the IB clients
options lnet networks="o2ib(ib0)"
options ko2iblnd concurrent_sends=7
options ptlrpc at_max=600 
options ost oss_num_threads=512

And modprobe.conf for the TCP clients
options lnet networks="tcp0(eth0)"
options ptlrpc at_max=600 
options ost oss_num_threads=512

The ''lctl list_nids'' command prints the expected results on
the servers and clients, listing the networks provided in the modprobe.conf file

I added the failover and mgsnode settings to each lun (6 luns) using the
following:
tunefs.lustre --failnode=172.20.20.31 at tcp --failnode=172.20.20.32 \
--mgsnode=172.20.20.30 at tcp /dev/mpath/lun1

With the final parameters being:
Persistent mount opts: errors=remount-ro,extents,mballoc
Parameters: failover.node=172.20.21.31 at o2ib failover.node=172.20.21.32 at
o2ib mgsnode=172.20.21.30 at o2ib failover.node=172.20.20.31 at tcp
failover.node=172.20.20.32 at tcp mgsnode=172.20.20.30 at tcp

The /etc/fstab on the TCP clients has this entry:
172.20.20.30 at tcp0:/lustre    /lustre                 lustre  _netdev        
0 0

I''ve rebooted all of the servers after making all of the changes and
still I can''t mount from the TCP clients, but can from the IB clients.

Any suggestions?

================================Mike Hanby
mhanby at uab.edu
Information Systems Specialist II
IT HPCS / Research Computing

David Dillow

2009-Oct-30 01:31 UTC

head link

[Lustre-discuss] Multihoned Problem, can mount o2ib but not tcp

On Thu, 2009-10-29 at 17:13 -0500, Mike Hanby wrote:> I added the failover and mgsnode settings to each lun (6 luns) using the
following:
> tunefs.lustre --failnode=172.20.20.31 at tcp --failnode=172.20.20.32 \
> --mgsnode=172.20.20.30 at tcp /dev/mpath/lun1
Did use --writeconf on the servers? You need to do so with LNET up with
the appropriate nids, so that clients (and the MDS) can find the servers
on both networks.

I think this is covered in the manual, so you should check there as
well. 
-- 
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office

Mike Hanby

2009-Oct-30 16:38 UTC

head link

[Lustre-discuss] Multihoned Problem, can mount o2ib but not tcp

No, I didn''t thanks for pointing out --writeconf.

I reran the tunefs.lustre on each of the luns (after stopping the heartbeat
service on each of the OSS nodes) using --writeconf:

tunefs.lustre --writeconf --erase-params \
 --param="failover.node=172.20.21.31 at o2ib" \
 --param="failover.node=172.20.21.32 at o2ib" \
 --param="mgsnode=172.20.21.30 at o2ib" \
 --param="failover.node=172.20.20.31 at tcp" \
 --param="failover.node=172.20.20.32 at tcp" \
 --param="mgsnode=172.20.20.30 at tcp" /dev/mpath/lun1

checking for existing Lustre data: found CONFIGS/mountdata
Reading CONFIGS/mountdata

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x2
              (OST )
Persistent mount opts: errors=remount-ro,extents,mballoc
Parameters: failover.node=172.20.21.31 at o2ib failover.node=172.20.21.32 at
o2ib mgsnode=172.20.21.30 at o2ib


   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x542
              (OST update writeconf )
Persistent mount opts: errors=remount-ro,extents,mballoc
Parameters: failover.node=172.20.21.31 at o2ib failover.node=172.20.21.32 at
o2ib mgsnode=172.20.21.30 at o2ib failover.node=172.20.20.31 at tcp
failover.node=172.20.20.32 at tcp mgsnode=172.20.20.30 at tcp

Writing CONFIGS/mountdata

Following fresh boots of the OSSes and MDS and the tcp client, I''m
still getting the same error:
kernel: LustreError: 2036:0:(events.c:460:ptlrpc_uuid_to_peer()) No NID found
for 172.20.21.30 at o2ib
kernel: LustreError: 2036:0:(client.c:69:ptlrpc_uuid_to_connection()) cannot
find peer 172.20.21.30 at o2ib!
kernel: LustreError: 2036:0:(ldlm_lib.c:329:client_obd_setup()) can''t
add initial connection
kernel: LustreError: 2036:0:(obd_config.c:370:class_setup()) setup
lustre-MDT0000-mdc-ffff81003f7c3400 failed (-2)
kernel: LustreError: 2036:0:(obd_config.c:1197:class_config_llog_handler()) Err
-2 on cfg command:
kernel: LustreError: 15c-8: MGC172.20.20.30 at tcp: The configuration from log
''lustre-client'' failed (-2). This may be the result of
communication errors between this node and the MGS, a bad configuration, or
other errors. See the syslog for more information.
kernel: LustreError: 1959:0:(llite_lib.c:1171:ll_fill_super()) Unable to process
log: -2
kernel: LustreError: 1959:0:(obd_config.c:441:class_cleanup()) Device 2 not
setup
kernel: LustreError: 1959:0:(ldlm_request.c:1030:ldlm_cli_cancel_req()) Got rc
-108 from cancel RPC: canceling anyway
kernel: LustreError: 1959:0:(ldlm_request.c:1533:ldlm_cli_cancel_list())
ldlm_cli_cancel_list: -108
kernel: LustreError: 1959:0:(obd_mount.c:1997:lustre_fill_super()) Unable to
mount  (-2)

Do I need to run tunefs.lustre on the MGS node as well?

Thanks, Mike

-----Original Message-----
From: David Dillow [mailto:dillowda at ornl.gov] 
Sent: Thursday, October 29, 2009 8:32 PM
To: Mike Hanby
Cc: ''lustre-discuss at lists.lustre.org''
Subject: Re: [Lustre-discuss] Multihoned Problem, can mount o2ib but not tcp

On Thu, 2009-10-29 at 17:13 -0500, Mike Hanby wrote:> I added the failover and mgsnode settings to each lun (6 luns) using the
following:
> tunefs.lustre --failnode=172.20.20.31 at tcp --failnode=172.20.20.32 \
> --mgsnode=172.20.20.30 at tcp /dev/mpath/lun1
Did use --writeconf on the servers? You need to do so with LNET up with
the appropriate nids, so that clients (and the MDS) can find the servers
on both networks.

I think this is covered in the manual, so you should check there as
well. 
-- 
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office

Andreas Dilger

2009-Oct-30 18:18 UTC

head link

[Lustre-discuss] Multihoned Problem, can mount o2ib but not tcp

On 2009-10-30, at 10:38, Mike Hanby wrote:> kernel: LustreError: 15c-8: MGC172.20.20.30 at tcp: The configuration  
> from log ''lustre-client'' failed (-2). This may be the
result of
> communication errors between this node and the MGS, a bad  
> configuration, or other errors. See the syslog for more information.
>
> Do I need to run tunefs.lustre on the MGS node as well?
Yes, since you specified multiple NIDs for the MGS, the MGS itself  
needs to
know to accept connections on that interface.
> Thanks, Mike
>
> -----Original Message-----
> From: David Dillow [mailto:dillowda at ornl.gov]
> Sent: Thursday, October 29, 2009 8:32 PM
> To: Mike Hanby
> Cc: ''lustre-discuss at lists.lustre.org''
> Subject: Re: [Lustre-discuss] Multihoned Problem, can mount o2ib but  
> not tcp
>
> On Thu, 2009-10-29 at 17:13 -0500, Mike Hanby wrote:
>> I added the failover and mgsnode settings to each lun (6 luns)  
>> using the following:
>> tunefs.lustre --failnode=172.20.20.31 at tcp --failnode=172.20.20.32 \
>> --mgsnode=172.20.20.30 at tcp /dev/mpath/lun1
>
> Did use --writeconf on the servers? You need to do so with LNET up  
> with
> the appropriate nids, so that clients (and the MDS) can find the  
> servers
> on both networks.
>
> I think this is covered in the manual, so you should check there as
> well.
> -- 
> Dave Dillow
> National Center for Computational Science
> Oak Ridge National Laboratory
> (865) 241-6602 office
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Mike Hanby

2009-Oct-30 18:30 UTC

head link

[Lustre-discuss] Multihoned Problem, can mount o2ib but not tcp

So, assuming the MGS is /dev/loop0 on my mds server, something like this
(without the --dryrun), also, I''ll need to umount /dev/loop0 first,
correct?:

tunefs.lustre --dryrun --writeconf --erase-params \
 --param="failover.node=172.20.21.30 at o2ib" \
 --param="failover.node=172.20.20.30 at tcp" \
 --param="mdt.group_upcall=NONE" /dev/loop0

Reading CONFIGS/mountdata

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x405
              (MDT MGS )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters: failover.node=172.20.21.30 at o2ib mdt.group_upcall=NONE


   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x545
              (MDT MGS update writeconf )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters: failover.node=172.20.21.30 at o2ib failover.node=172.20.20.30 at tcp
mdt.group_upcall=NONE

exiting before disk write.

-----Original Message-----
From: Andreas.Dilger at sun.com [mailto:Andreas.Dilger at sun.com] On Behalf Of
Andreas Dilger
Sent: Friday, October 30, 2009 1:18 PM
To: Mike Hanby
Cc: ''David Dillow''; ''lustre-discuss at
lists.lustre.org''
Subject: Re: [Lustre-discuss] Multihoned Problem, can mount o2ib but not tcp

On 2009-10-30, at 10:38, Mike Hanby wrote:> kernel: LustreError: 15c-8: MGC172.20.20.30 at tcp: The configuration  
> from log ''lustre-client'' failed (-2). This may be the
result of
> communication errors between this node and the MGS, a bad  
> configuration, or other errors. See the syslog for more information.
>
> Do I need to run tunefs.lustre on the MGS node as well?
Yes, since you specified multiple NIDs for the MGS, the MGS itself  
needs to
know to accept connections on that interface.
> Thanks, Mike
>
> -----Original Message-----
> From: David Dillow [mailto:dillowda at ornl.gov]
> Sent: Thursday, October 29, 2009 8:32 PM
> To: Mike Hanby
> Cc: ''lustre-discuss at lists.lustre.org''
> Subject: Re: [Lustre-discuss] Multihoned Problem, can mount o2ib but  
> not tcp
>
> On Thu, 2009-10-29 at 17:13 -0500, Mike Hanby wrote:
>> I added the failover and mgsnode settings to each lun (6 luns)  
>> using the following:
>> tunefs.lustre --failnode=172.20.20.31 at tcp --failnode=172.20.20.32 \
>> --mgsnode=172.20.20.30 at tcp /dev/mpath/lun1
>
> Did use --writeconf on the servers? You need to do so with LNET up  
> with
> the appropriate nids, so that clients (and the MDS) can find the  
> servers
> on both networks.
>
> I think this is covered in the manual, so you should check there as
> well.
> -- 
> Dave Dillow
> National Center for Computational Science
> Oak Ridge National Laboratory
> (865) 241-6602 office
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Mike Hanby

2009-Oct-30 21:16 UTC

head link

[Lustre-discuss] Multihoned Problem, can mount o2ib but not tcp

Btw, subject should read "Multihomed..." I always mistype that for
some reason.

Anyhow, both networks are now working. Thanks to both for the clues, writeconf
and adding to the mgs.

Mike

-----Original Message-----
From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces
at lists.lustre.org] On Behalf Of Mike Hanby
Sent: Friday, October 30, 2009 1:31 PM
To: ''lustre-discuss at lists.lustre.org''
Subject: Re: [Lustre-discuss] Multihoned Problem, can mount o2ib but not tcp

So, assuming the MGS is /dev/loop0 on my mds server, something like this
(without the --dryrun), also, I''ll need to umount /dev/loop0 first,
correct?:

tunefs.lustre --dryrun --writeconf --erase-params \
 --param="failover.node=172.20.21.30 at o2ib" \
 --param="failover.node=172.20.20.30 at tcp" \
 --param="mdt.group_upcall=NONE" /dev/loop0

Reading CONFIGS/mountdata

   Read previous values:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x405
              (MDT MGS )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters: failover.node=172.20.21.30 at o2ib mdt.group_upcall=NONE


   Permanent disk data:
Target:     lustre-MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x545
              (MDT MGS update writeconf )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters: failover.node=172.20.21.30 at o2ib failover.node=172.20.20.30 at tcp
mdt.group_upcall=NONE

exiting before disk write.

-----Original Message-----
From: Andreas.Dilger at sun.com [mailto:Andreas.Dilger at sun.com] On Behalf Of
Andreas Dilger
Sent: Friday, October 30, 2009 1:18 PM
To: Mike Hanby
Cc: ''David Dillow''; ''lustre-discuss at
lists.lustre.org''
Subject: Re: [Lustre-discuss] Multihoned Problem, can mount o2ib but not tcp

On 2009-10-30, at 10:38, Mike Hanby wrote:> kernel: LustreError: 15c-8: MGC172.20.20.30 at tcp: The configuration  
> from log ''lustre-client'' failed (-2). This may be the
result of
> communication errors between this node and the MGS, a bad  
> configuration, or other errors. See the syslog for more information.
>
> Do I need to run tunefs.lustre on the MGS node as well?
Yes, since you specified multiple NIDs for the MGS, the MGS itself  
needs to
know to accept connections on that interface.
> Thanks, Mike
>
> -----Original Message-----
> From: David Dillow [mailto:dillowda at ornl.gov]
> Sent: Thursday, October 29, 2009 8:32 PM
> To: Mike Hanby
> Cc: ''lustre-discuss at lists.lustre.org''
> Subject: Re: [Lustre-discuss] Multihoned Problem, can mount o2ib but  
> not tcp
>
> On Thu, 2009-10-29 at 17:13 -0500, Mike Hanby wrote:
>> I added the failover and mgsnode settings to each lun (6 luns)  
>> using the following:
>> tunefs.lustre --failnode=172.20.20.31 at tcp --failnode=172.20.20.32 \
>> --mgsnode=172.20.20.30 at tcp /dev/mpath/lun1
>
> Did use --writeconf on the servers? You need to do so with LNET up  
> with
> the appropriate nids, so that clients (and the MDS) can find the  
> servers
> on both networks.
>
> I think this is covered in the manual, so you should check there as
> well.
> -- 
> Dave Dillow
> National Center for Computational Science
> Oak Ridge National Laboratory
> (865) 241-6602 office
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

Lustre discuss - Oct 2009 - Multihoned Problem, can mount o2ib but not tcp

[Lustre-discuss] Multihoned Problem, can mount o2ib but not tcp

[Lustre-discuss] Multihoned Problem, can mount o2ib but not tcp

[Lustre-discuss] Multihoned Problem, can mount o2ib but not tcp

[Lustre-discuss] Multihoned Problem, can mount o2ib but not tcp

[Lustre-discuss] Multihoned Problem, can mount o2ib but not tcp

[Lustre-discuss] Multihoned Problem, can mount o2ib but not tcp