I have two boxes that have this: [root at lustrethree Desktop]# ifconfig eth0 Link encap:Ethernet HWaddr 00:1B:21:2A:17:76 inet addr:192.168.0.19 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::21b:21ff:fe2a:1776/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1501960 errors:0 dropped:0 overruns:0 frame:0 TX packets:3792561 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:120168321 (114.6 MiB) TX bytes:5300070662 (4.9 GiB) Base address:0xec00 Memory:febe0000-fec00000 eth1 Link encap:Ethernet HWaddr 00:1B:21:2A:1C:DC inet addr:192.168.0.20 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::21b:21ff:fe2a:1cdc/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:828283 errors:0 dropped:0 overruns:0 frame:0 TX packets:12 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:55673426 (53.0 MiB) TX bytes:846 (846.0 b) Base address:0xe880 Memory:feb80000-feba0000 eth2 Link encap:Ethernet HWaddr 00:22:15:06:3A:0F inet addr:192.168.0.21 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::222:15ff:fe06:3a0f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:828047 errors:0 dropped:0 overruns:0 frame:0 TX packets:11 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:55657196 (53.0 MiB) TX bytes:782 (782.0 b) Interrupt:185 eth3 Link encap:Ethernet HWaddr 00:22:15:06:3A:10 inet addr:192.168.0.22 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::222:15ff:fe06:3a10/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:827857 errors:0 dropped:0 overruns:0 frame:0 TX packets:11 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:55644744 (53.0 MiB) TX bytes:782 (782.0 b) Interrupt:209 eth4 Link encap:Ethernet HWaddr 00:22:15:06:3A:11 inet addr:192.168.0.23 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::222:15ff:fe06:3a11/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:827706 errors:0 dropped:0 overruns:0 frame:0 TX packets:11 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:55634788 (53.0 MiB) TX bytes:782 (782.0 b) Interrupt:169 eth5 Link encap:Ethernet HWaddr 00:22:15:06:3A:12 inet addr:192.168.0.24 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::222:15ff:fe06:3a12/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:827519 errors:0 dropped:0 overruns:0 frame:0 TX packets:11 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:55622528 (53.0 MiB) TX bytes:782 (782.0 b) Interrupt:193 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:1485135 errors:0 dropped:0 overruns:0 frame:0 TX packets:1485135 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:5089873659 (4.7 GiB) TX bytes:5089873659 (4.7 GiB) [root at lustrethree Desktop]# Would it be better to have these two boxes as OSS''s or as MDT or MGS machines? Currently they are configured 1 as a MGS and the other as the MDT. The question is does LNET use the available tcp0 connections different from the OSS perspective as opposed to the MDT or MGS perspective?
> I have two boxes that have this:> [root at lustrethree Desktop]# ifconfig > eth0 Link encap:Ethernet HWaddr 00:1B:21:2A:17:76 > inet addr:192.168.0.19 Bcast:192.168.0.255 Mask:255.255.255.0 > RX bytes:120168321 (114.6 MiB) TX bytes:5300070662 (4.9 GiB) > [ ... ] > eth1 Link encap:Ethernet HWaddr 00:1B:21:2A:1C:DC > inet addr:192.168.0.20 Bcast:192.168.0.255 Mask:255.255.255.0 > RX bytes:55673426 (53.0 MiB) TX bytes:846 (846.0 b) > [ ... another 4 like that, 192.168.0.21-24 ... ]That''s a very bizarre network configuration, you have 5 interfaces on the same subnet (presumably all plugged into the same switch) with no load balancing, as all the outgoing traffic goes via ''eth0''. You have some better alternatives: * Use bonding (if the switch supports to ties together the 5 interfaces as one virtual interface with a single IP address. * Use something like ''nexthop'' routing (and a couple other tricks) to split the load across the several interfaces. This is easier for the outgoing traffic than the incoming traffic, but it seems you have a lot more outgoing traffic. * Use 1 10Gb/s card per server and a 1Gb/s switch with 2 10Gb/s ports. 10Gb/s cards and switches have fallen in price a lot recently (check Myri.com), and a server that can do several hundred MB/s really deserves a nice 10Gb/s interface. IIRC ''lnet'' has something like bonding built in, but I am not sure that it handles multiple addresses in the same subnet well.> Would it be better to have these two boxes as OSS''s or as MDT > or MGS machines? Currently they are configured 1 as a MGS and > the other as the MDT.If these are the two servers with gigantic disk arrays, I''d have on each both MDS and OSS. Possibly with the OSTs replicated across both machines in an active/passive configuration.> The question is does LNET use the available tcp0 connections > different from the OSS perspective as opposed to the MDT or > MGS perspective?Not sure that the question means.
Hi folks, Lustre doesn''t support any inherent link aggregation, it simply utilizes the device node the OS presents. If this is a bonded NIC, it will use it no problem, but the underlying device driver takes care of load balancing and distribution. I''ve used Lustre 1.6.x quite successfully with load-balanced 802.3ad configurations; in some of my tests I was able to get about 350 MB/s aggregate sustained across two OSS nodes with 2 x GigE bonded each. 802.3ad link aggregation is a standard NIC bonding protocol, and is supported on all good quality L3 switches and by vendors like Cisco, Foundry, Extreme, and Juniper. cheers, Klaus On 1/11/09 11:37 AM, "Peter Grandi" <pg_lus at lus.for.sabi.co.UK> etched on stone tablets:>> I have two boxes that have this: > >> [root at lustrethree Desktop]# ifconfig >> eth0 Link encap:Ethernet HWaddr 00:1B:21:2A:17:76 >> inet addr:192.168.0.19 Bcast:192.168.0.255 Mask:255.255.255.0 >> RX bytes:120168321 (114.6 MiB) TX bytes:5300070662 (4.9 GiB) >> [ ... ] >> eth1 Link encap:Ethernet HWaddr 00:1B:21:2A:1C:DC >> inet addr:192.168.0.20 Bcast:192.168.0.255 Mask:255.255.255.0 >> RX bytes:55673426 (53.0 MiB) TX bytes:846 (846.0 b) >> [ ... another 4 like that, 192.168.0.21-24 ... ] > > That''s a very bizarre network configuration, you have 5 > interfaces on the same subnet (presumably all plugged into the > same switch) with no load balancing, as all the outgoing traffic > goes via ''eth0''. > > You have some better alternatives: > > * Use bonding (if the switch supports to ties together the 5 > interfaces as one virtual interface with a single IP address. > > * Use something like ''nexthop'' routing (and a couple other > tricks) to split the load across the several interfaces. This > is easier for the outgoing traffic than the incoming traffic, > but it seems you have a lot more outgoing traffic. > > * Use 1 10Gb/s card per server and a 1Gb/s switch with 2 10Gb/s > ports. 10Gb/s cards and switches have fallen in price a lot > recently (check Myri.com), and a server that can do several > hundred MB/s really deserves a nice 10Gb/s interface. > > IIRC ''lnet'' has something like bonding built in, but I am not > sure that it handles multiple addresses in the same subnet well. > >> Would it be better to have these two boxes as OSS''s or as MDT >> or MGS machines? Currently they are configured 1 as a MGS and >> the other as the MDT. > > If these are the two servers with gigantic disk arrays, I''d have > on each both MDS and OSS. Possibly with the OSTs replicated > across both machines in an active/passive configuration. > >> The question is does LNET use the available tcp0 connections >> different from the OSS perspective as opposed to the MDT or >> MGS perspective? > > Not sure that the question means. > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
[snip]> > IIRC ''lnet'' has something like bonding built in, but I am not > sure that it handles multiple addresses in the same subnet well. > >Not anymore! Kernel bonding is the preferred method, LNET no longer bonds ethernet interfaces, as the kernel bonding is far superior. ( We only did bonding in LNET when kernel bonding was somewhat broken, that was several years ago. Kernel bonding is fine now) cliffw