Lustre devel - Jan 2007 - [Bug 11555] how to configure lustre with two tcp subnet

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=11555

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Priority|P3                          |P2


change priority to p2.

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=11555

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Group|                            |Enterprise Support_CHEVRON


change group

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=11555



Created an attachment (id=9347)
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
 --> (https://bugzilla.lustre.org/attachment.cgi?id=9347&action=view)
Make_xml

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=11555



I attached the make_xml

and
>From the client
[root@hal1400 ~]#  mount -t lustre 146.36.70.141@tcp1:/mds001/client /cpfs/fs2
mount.lustre: mount(146.36.70.141@tcp1:/mds001/client, /cpfs/fs2) failed: No 
such file or directory
mds nid 0:       ngsmds1b@tcp1
mds name:        mds001
profile:         client
options:         rw

[root@hal1400 ~]# lctl ping 146.36.70.141
failed to ping ngsmds1b@tcp: Input/output error

[root@hal1400 ~]# lctl ping 146.36.70.141@tcp1
failed to ping ngsmds1b@tcp1: No route to host

[root@hal1400 ~]# lctl list_nids
hal1400@tcp
>From the mds, it didn''t pick up the the tcp1 as 146.36.70.141.
[root@ngsmds1 ~]# lctl list_nids
146.36.104.21@tcp
146.36.104.21@tcp1
[root@ngsmds1 ~]# lctl ping 146.36.71.181@tcp0
failed to ping 146.36.71.181@tcp: Protocol error
[root@ngsmds1 ~]# lctl ping 146.36.71.181@tcp1
failed to ping 146.36.71.181@tcp1: Input/output error

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=11555



ok, I modified the modprobe.conf now

from the MDS and OSS
 I have
  [root@ngsmds1 ~]# lctl list_nids
146.36.104.21@tcp
146.36.70.141@tcp1


The problem for the client nodes is we have two lustre running. 
Lustre one (old one) has one network tcp0
the lustre two (new one) has two network tcp0 and tcp1 now.

for the nodes they go to tcp0 they are ok. I can mount them.
for the nodes they go to tcp0 for Lustre one, and tcp1 for Lustre two. What 
should I put in modprobe.conf ?

The client nodes only have one interface eth0. 
So I tried 
   options lent networks=tcp0(eth0), tcp1(eth0)

Now I can''t mount lustre one and lustre two ?

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=11555



cliff, 
old lustre is mounted on /cpfs/fs1 with ngs2mds1 the mds name
new lustre will be mount on /cpfs/fs2 with ngsmds1 as the mds name 

Now
after I modified the modprobe.conf I can

[root@hal1403 ~]# mount -t lustre ngsmds1:/mds001/client /cpfs/fs2
[root@hal1403 ~]# mount |grep cpfs
config on /cpfs/fs1 type lustre_lite 
(rw,osc=lov001,mdc=MDC_hal1403_mds2001_MNT_client)
ngsmds1:/mds001/client on /cpfs/fs2 type lustre (rw)

That is good however If I umount the old Lustre 

[root@hal1403 ~]# umount /cpfs/fs1

 and tried to remount the old lustre by 
[root@hal1403 ~]#  mount -t lustre ngs2mds1:/mds001/client /cpfs/fs1
mount.lustre: mount(ngs2mds1:/mds001/client, /cpfs/fs1) failed: Input/output 
error
mds nid 0:       ngs2mds1@tcp
mds name:        mds001
profile:         client
options:         rw

It failed.  I can''t ping the new lustre or old lustre by

[root@hal1403 ~]# lctl ping 146.36.104.21@tcp0
failed to ping ngsmds1@tcp: Connection timed out
[root@hal1403 ~]# lctl ping 146.36.104.21@tcp1
failed to ping ngsmds1@tcp1: No route to host

But new lustre is still mounted.

In the OLD lustre mds log
I have

Jan 18 12:45:01 ngs2mds1 crond(pam_unix)[24688]: session opened for user root 
by (uid=0)
Jan 18 12:45:01 ngs2mds1 crond(pam_unix)[24688]: session closed for user root
Jan 18 12:45:40 ngs2mds1 kernel: LustreError: 4433:0:
(ldlm_lib.c:541:target_handle_connect()) @@@ UUID ''mds001'' is
not available
for connect (no target) req@0000010222582400 x261416/t0
o38-><?>@<?>:-1 lens
240/0 ref 0 fl Interpret:/0/0 rc 0/0
Jan 18 12:45:40 ngs2mds1 kernel: LustreError: 4433:0:
(ldlm_lib.c:1288:target_send_reply_msg()) @@@ processing error (-19) 
req@0000010222582400 x261416/t0 o38-><?>@<?>:-1 lens 240/0 ref 0
fl
Interpret:/0/0 rc -19/0

However I can umount and remount the new lustre now?

[root@hal1403 ~]#  umount /cpfs/fs2
[root@hal1403 ~]#  mount -t lustre ngsmds1:/mds001/client /cpfs/fs2

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=11555



ok, we will rename it and do a reread. 

however I found that if I lconf it, it works.   So no problem now.

[root@hal1400 /]# mount -t lustre ngs2mds1:/mds001/client /cpfs/fs1
mount.lustre: mount(ngs2mds1:/mds001/client, /cpfs/fs1) failed: Input/output 
error
mds nid 0:       ngs2mds1@tcp
mds name:        mds001
profile:         client
options:         rw
[root@hal1400 /]# lconf --node client /etc/lustre/config.xml
MDC: MDC_hal1400_mds2001_MNT_client 677e8_MNT_client_0ce069cf1e mds2001_UUID
MDC: MDC_hal1400_mds2001_MNT_client 677e8_MNT_client_0ce069cf1e 
MTPT: MNT_client MNT_client_UUID /cpfs/fs1 mds2001_UUID lov001_UUID
[root@hal1400 /]# ls /cpfs/fs1
chap  data  iozone  temp