Hello, We are expanding our Lustre over 10GbE TCP setup. We are going to add few more OSSes and another 10GbE switch because we need more ports. All OSSes and MDS have 2 x 10GbE interfaces in bonding-alb (same for clients). For the new setup we have few choices to connect switches: 1. Just add a new switch move 2nd interfaces from all servers to the new switch and reconfigure all clients and servers to use arp ping. (maybe we will need to switch bonding to balance-rr?, will test it). The scary part is that the network noise will increase with increasing number of clients and servers, so I would prefer to keep MII monitoring. 2. Setup trunking on switches, connect them with 4 x 10GbE lines, move 2nd server interfaces to the new switch. In this case we can leave bonding-alb and mii monitoring. For me it looks like 2nd option should be better way to go. I expect with the alb mode Lustre clients and servers will be able to go over the other switch in case connection to local switch is to busy. And should be able to keep fs online if one switch goes down. Is there any recommended setup for Lnet how to connect switches? Maybe someone already have experience? The basic requirement is maximum throughput and to be able to have access to the filesystem in case one switch goes down. Thanks, Alex.
You are running into a similar problem that comes from even growing a SAN by adding fiber switches. Option 2 is a good way to go if you can afford the ports. I would still consider it a stop-gap until you are able to purchase a more powerful core switch with enough ports. Otherwise, once you outgrow this setup, you will end up with several little islands of switches that are throttled between them, no matter how you lay it out. We experienced the same thing, although we were running infiniband for the backend. Our cluster outgrew the infiniband switch, so we got another and just dealt with it until we were able to procure a larger one. Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California voice: 831-656-6238 -----Original Message----- From: lustre-discuss-bounces-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org [mailto:lustre-discuss-bounces-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org] On Behalf Of Alexander Oltu Sent: Monday, July 08, 2013 6:42 AM To: lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org Subject: [Lustre-discuss] LNET over 2 x 10 GbE switches Hello, We are expanding our Lustre over 10GbE TCP setup. We are going to add few more OSSes and another 10GbE switch because we need more ports. All OSSes and MDS have 2 x 10GbE interfaces in bonding-alb (same for clients). For the new setup we have few choices to connect switches: 1. Just add a new switch move 2nd interfaces from all servers to the new switch and reconfigure all clients and servers to use arp ping. (maybe we will need to switch bonding to balance-rr?, will test it). The scary part is that the network noise will increase with increasing number of clients and servers, so I would prefer to keep MII monitoring. 2. Setup trunking on switches, connect them with 4 x 10GbE lines, move 2nd server interfaces to the new switch. In this case we can leave bonding-alb and mii monitoring. For me it looks like 2nd option should be better way to go. I expect with the alb mode Lustre clients and servers will be able to go over the other switch in case connection to local switch is to busy. And should be able to keep fs online if one switch goes down. Is there any recommended setup for Lnet how to connect switches? Maybe someone already have experience? The basic requirement is maximum throughput and to be able to have access to the filesystem in case one switch goes down. Thanks, Alex. _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Brian, thanks! The new switch has double amount of ports. Based on your suggestion we will migrate all connections to the new one and still there should be plenty available for later file system expansion. Alex. On Tue, 9 Jul 2013 06:07:02 +0000 Andrus, Brian Contractor wrote:> You are running into a similar problem that comes from even growing a > SAN by adding fiber switches. Option 2 is a good way to go if you can > afford the ports. I would still consider it a stop-gap until you are > able to purchase a more powerful core switch with enough ports. > Otherwise, once you outgrow this setup, you will end up with several > little islands of switches that are throttled between them, no matter > how you lay it out. > > We experienced the same thing, although we were running infiniband > for the backend. Our cluster outgrew the infiniband switch, so we got > another and just dealt with it until we were able to procure a larger > one. > > > Brian Andrus > ITACS/Research Computing > Naval Postgraduate School > Monterey, California > voice: 831-656-6238 > > > > -----Original Message----- > From: lustre-discuss-bounces-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org > [mailto:lustre-discuss-bounces-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org] On Behalf Of > Alexander Oltu Sent: Monday, July 08, 2013 6:42 AM To: > lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org Subject: [Lustre-discuss] LNET over 2 > x 10 GbE switches > > Hello, > > We are expanding our Lustre over 10GbE TCP setup. We are going to > add few more OSSes and another 10GbE switch because we need more > ports. > > All OSSes and MDS have 2 x 10GbE interfaces in bonding-alb (same for > clients). > > For the new setup we have few choices to connect switches: > > 1. Just add a new switch move 2nd interfaces from all servers to the > new switch and reconfigure all clients and servers to use arp ping. > (maybe we will need to switch bonding to balance-rr?, will test it). > The scary part is that the network noise will increase with > increasing number of clients and servers, so I would prefer to keep > MII monitoring. > > > 2. Setup trunking on switches, connect them with 4 x 10GbE lines, > move 2nd server interfaces to the new switch. In this case we can > leave bonding-alb and mii monitoring. > > For me it looks like 2nd option should be better way to go. I expect > with the alb mode Lustre clients and servers will be able to go over > the other switch in case connection to local switch is to busy. And > should be able to keep fs online if one switch goes down. > > > Is there any recommended setup for Lnet how to connect switches? > Maybe someone already have experience? > > The basic requirement is maximum throughput and to be able to have > access to the filesystem in case one switch goes down. > > Thanks, > Alex. > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss