Hi, I'm always confused of which NID to use if multiple LNET interfaces are available on server and client. Someone told me connection between Lustre client and OSS is determined by which NID of MGS is specified when mounting To make it clear, I establish a VM environment to verify: In the testing environment, there are only one MDS, OSS and Client. Each of them has two Ethernet interfaces. All nodes have the following lnet configuration in modprobe.d options lnet networks=tcp0(eth0),tcp1(eth1) [root@mds1 ~]# lctl list_nids 192.168.122.240@tcp 192.168.100.100@tcp1 [root@oss1 ~]# lctl list_nids 192.168.122.194@tcp 192.168.100.101@tcp1 [root@client ~]# lctl list_nids 192.168.122.70@tcp 192.168.100.102@tcp1 [root@mds1 ~]# mkfs.lustre --mgs --mdt --fsname=data --index=0 --reformat /dev/sda When formatting OST, tcp0 is used to establish the connection between OST and MGT. [root@oss1 ~]# mkfs.lustre --ost --mgsnode=192.168.122.240@tcp0 --fsname=data --index=0 --reformat /dev/sda On Lustre client, I intentionally mount it with tcp1 [root@client ~]# mount | grep lustre 192.168.100.100@tcp1:/data on /lustre type lustre (rw) Now I dd a file on Lustre filesystem, you can see that tcp0 is used when writing on OST. Why? [root@client lustre]# ifconfig eth0 | grep TX TX packets:224400 errors:0 dropped:0 overruns:0 carrier:0 RX bytes:4450732 (4.2 MiB) TX bytes:1064624772 (1015.3 MiB) [root@client lustre]# dd if=/dev/zero of=testfile bs=1M count=500 [root@client lustre]# ifconfig eth0 | grep TX TX packets:337851 errors:0 dropped:0 overruns:0 carrier:0 RX bytes:5578294 (5.3 MiB) TX bytes:1596746050 (1.4 GiB) Regards, Patrick _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss