Dear All, I need to buy a bigger GigE switch for my GlusterFS cluster and I am trying to decide whether or not a much more expensive one would be justified. I have limited experience with networking so I don't know if it would be appropriate to spend ?500, ?1500 or ?3500 for a 48-port switch. Those rough costs are based on a comparison of 3 Dell Powerconnect switches: the 5548 (bigger version of what we have now), the 6248 and the 7048. The servers in the cluster are nothing special - mostly Supermicro with SATA drives and 1GigE network adapters. I can only justify spending more than ~?500 if I can be sure that users would notice the difference. Some of the users' applications do lots of small reads and writes, and they do run much more slowly if all the servers are not connected to the same switch, as is the case now while I don't have a big enough switch. Any advice or comments would be much appreciated. Regards Dan.
You could get a cisco switch that supports cut through instead of store-and-forward, for lower latency. Other than that, compare the port to port forwarding times and see if there is a difference between the switches you are looking at (probably not) and make your decision based on that. Consider connecting everything to two switches, for failover in case a switch breaks? On 1/27/2012 2:04 PM, Dan Bretherton wrote:> Dear All, > I need to buy a bigger GigE switch for my GlusterFS cluster and I am > trying to decide whether or not a much more expensive one would be > justified. I have limited experience with networking so I don't know > if it would be appropriate to spend ?500, ?1500 or ?3500 for a 48-port > switch. Those rough costs are based on a comparison of 3 Dell > Powerconnect switches: the 5548 (bigger version of what we have now), > the 6248 and the 7048. The servers in the cluster are nothing special > - mostly Supermicro with SATA drives and 1GigE network adapters. I > can only justify spending more than ~?500 if I can be sure that users > would notice the difference. Some of the users' applications do lots > of small reads and writes, and they do run much more slowly if all the > servers are not connected to the same switch, as is the case now while > I don't have a big enough switch. Any advice or comments would be > much appreciated. > > Regards > Dan. > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
If you are considering as much as ?3500 for a switch, you might want to consider infiniband qdr instead. We don't currently have it here, but are considering it. From what I can it has lower latency, can do 40gbps, is reasonably priced (slightly better than 10gbe, not comaring directly to gb) . That said, you would also have to budget for cards and cables even though the switch price by itself is not so bad... As you have multiple switches (sounds like 5524 or maybe 5424), the first next to 0 cost change you should do (if not already done) is setup a LAG between the switches and run 2-4 cables between switches instead of 1.> -----Original Message----- > From: gluster-users-bounces at gluster.org [mailto:gluster-users- > bounces at gluster.org] On Behalf Of Dan Bretherton > Sent: Friday, January 27, 2012 8:05 AM > To: gluster-users > Subject: [Gluster-users] Switch recommendations > > Dear All, > I need to buy a bigger GigE switch for my GlusterFS cluster and I am > trying to decide whether or not a much more expensive one would be > justified. I have limited experience with networking so I don't know if > it would be appropriate to spend ?500, ?1500 or ?3500 for a 48-port > switch. Those rough costs are based on a comparison of 3 Dell > Powerconnect switches: the 5548 (bigger version of what we have now), > the 6248 and the 7048. The servers in the cluster are nothing special - > mostly Supermicro with SATA drives and 1GigE network adapters. I can > only justify spending more than ~?500 if I can be sure that users would > notice the difference. Some of the users' applications do lots of small > reads and writes, and they do run much more slowly if all the servers > are not connected to the same switch, as is the case now while I don't > have a big enough switch. Any advice or comments would be much > appreciated. > > Regards > Dan. > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
On Fri, Jan 27, 2012 at 01:04:39PM +0000, Dan Bretherton wrote:> I need to buy a bigger GigE switch for my GlusterFS cluster and I am > trying to decide whether or not a much more expensive one would be > justified. I have limited experience with networking so I don't > know if it would be appropriate to spend ?500, ?1500 or ?3500 for a > 48-port switch. Those rough costs are based on a comparison of 3 > Dell Powerconnect switches: the 5548 (bigger version of what we have > now), the 6248 and the 7048.If you already have 5524s then they are apparently stackable using a HDMI cable, have you tried that? I have bad experience with Powerconnect 5524's, but maybe you're just not tickling the same bugs I found (e.g. multicast problems, failing to learn MAC addresses) Haven't had these same problems with Netgear GSM7224V2, but then maybe they have a different set of bugs :-)> Some of the users' > applications do lots of small reads and writes, and they do run much > more slowly if all the servers are not connected to the same switch, > as is the case now while I don't have a big enough switch.If it's consistently a little slower for each individual file read and written, then it sounds like this usage pattern is extremely latency-sensitive. If it seems to be in bursts then maybe you are getting small amounts of packet loss, which is causing TCP to back off. Side note: if you are latency-sensitive then don't even think about using 10Gig over RJ45 - it actually has a higher latency than 1G. But 10Gig over SFP+ is fine. Regards, Brian.
Thanks for the advice Peter,> You could get a cisco switch that supports cut through instead of > store-and-forward, for lower latency.Interesting option, but those cost ~?10K and are out of our price range unfortunately. The "cut through" switching technology is also available in Dell's new Force 10 range I believe.> Other than that, compare the port to port forwarding times and see if > there is a difference between the switches you are looking at (probably > not) and make your decision based on that.You're right, there isn't much of a difference between the forwarding rate of the 5548 and the 6248, both are about 100Mpps. They also have a similar bandwidth of about 180Gbps. However the 7048 does better on both measures, with forwarding rate of 160Mpps and a bandwidth of 224Gbps. Unfortunately the 7048 costs five times as much as the 5548, and I don't know if the users would notice any difference at all. I expect some would and some wouldn't. -Dan. On 01/27/2012 01:48 PM, gluster-users-request at gluster.org wrote:> Date: Fri, 27 Jan 2012 14:16:36 +0100 > From: Peter Linder<peter.linder at fiberdirekt.se> > Subject: Re: [Gluster-users] Switch recommendations > To:gluster-users at gluster.org > Message-ID:<4F22A3B4.8000008 at fiberdirekt.se> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > You could get a cisco switch that supports cut through instead of > store-and-forward, for lower latency. > > Other than that, compare the port to port forwarding times and see if > there is a difference between the switches you are looking at (probably > not) and make your decision based on that. Consider connecting > everything to two switches, for failover in case a switch breaks? > > On 1/27/2012 2:04 PM, Dan Bretherton wrote: >> > Dear All, >> > I need to buy a bigger GigE switch for my GlusterFS cluster and I am >> > trying to decide whether or not a much more expensive one would be >> > justified. I have limited experience with networking so I don't know >> > if it would be appropriate to spend ?500, ?1500 or ?3500 for a 48-port >> > switch. Those rough costs are based on a comparison of 3 Dell >> > Powerconnect switches: the 5548 (bigger version of what we have now), >> > the 6248 and the 7048. The servers in the cluster are nothing special >> > - mostly Supermicro with SATA drives and 1GigE network adapters. I >> > can only justify spending more than ~?500 if I can be sure that users >> > would notice the difference. Some of the users' applications do lots >> > of small reads and writes, and they do run much more slowly if all the >> > servers are not connected to the same switch, as is the case now while >> > I don't have a big enough switch. Any advice or comments would be >> > much appreciated. >> > >> > Regards >> > Dan.
2012/1/27 Dan Bretherton <d.a.bretherton at reading.ac.uk>:> Dear All, > I need to buy a bigger GigE switch for my GlusterFS cluster and I am trying > to decide whether or not a much more expensive one would be justified. ?I > have limited experience with networking so I don't know if it would be > appropriate to spend ?500, ?1500 or ?3500 for a 48-port switch.Are there some network admins or someone with networking experience you could talk to? Do you use the management functions that your current switches support?> Those rough > costs are based on a comparison of 3 Dell Powerconnect switches: the 5548 > (bigger version of what we have now), the 6248 and the 7048. ?The servers in > the cluster are nothing special - mostly Supermicro with SATA drives and > 1GigE network adapters.How many server and clients do you have now? Do you plan to increase the numbers in the near future? In that case I'd suggest to get stackable switches for easier expansion. I can only justify spending more than ~?500 if I> can be sure that users would notice the difference. ?Some of the users' > applications do lots of small reads and writes, and they do run much more > slowly if all the servers are not connected to the same switch, as is the > case now while I don't have a big enough switch.How are the servers and clients distributed to the switches now? How are the switches connected to each other? Can you tell where your bottleneck is? Is it the connection between your switches or is it something else? Could you plug all the servers and some of the clients into one switch and the rest of the clients into the other switch(es) for a short period of time? There should be a big difference in speed between the two groups of clients. How does that compare to the speed you have now? Any advice or comments> would be much appreciated.What happens if one switch fails? Can the remaining equipment do some useful work until a replacement arrives? Depending on the answers it might be better to have two or more smaller stackable switches instead of one big switch, even if the big one might be cheaper. I don't have much experience with network administration so I cannot recommend a brand or type of switch. I just looked at the Netgear website and googled for prices: Two Netgear GS724TS stackable switches seem to cost nearly the same as one GS748TS, both are supposed to have "a 20 Gbps, dual-ring, highly redundant stacking bus". Regards, Harald