We have a cluster with QDR infiniband. Lustre MDS & OSS servers are using mellanox HCA while clients use qlogic HCAs. I believe the switches are qlogic QDR 12000 series. We are using ofed distributed with RHEL 6.2. We currently use the qlogic IB fabric manager (subnet manager). We have heard qlogic_fm is good for MPI but bad for lustre. We have the option of running ofed opensm. Does anyone have any experience/opinions about using qlogic fabric manager versus opensm ? thanks, chris hunter yale hpc group
The primary difference where Lustre is concerned is that Mellanox does RDMA in hardware, and QLogic did RDMA in software. Doesn''t really matter which subnet manager you run. -Marc ---- D. Marc Stearman Lustre Operations Lead stearman2-i2BcT+NCU+M@public.gmane.org 925.423.9670 On Jul 12, 2013, at 7:34 AM, Chris Hunter <chris.hunter-LrD5EImo2rg@public.gmane.org> wrote:> We have a cluster with QDR infiniband. Lustre MDS & OSS servers are > using mellanox HCA while clients use qlogic HCAs. I believe the switches > are qlogic QDR 12000 series. We are using ofed distributed with RHEL 6.2. > > We currently use the qlogic IB fabric manager (subnet manager). We have > heard qlogic_fm is good for MPI but bad for lustre. We have the option > of running ofed opensm. Does anyone have any experience/opinions about > using qlogic fabric manager versus opensm ? > > thanks, > chris hunter > yale hpc group > _______________________________________________ > Lustre-community mailing list > Lustre-community-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org > http://lists.lustre.org/mailman/listinfo/lustre-community
Thanks for the response. Qlogic & Mellanox claim to do a better job with routing on large/complex networks than opensm. They use buzz words such as "Dispersive Routing" or "Adaptive Routing". Is there any substance to these statements ? regards, chris hunter yale hpc group On 07/12/2013 06:34 PM, Stearman, Marc wrote:> The primary difference where Lustre is concerned is that Mellanox does RDMA in hardware, and QLogic did RDMA in software. Doesn''t really matter which subnet manager you run. > > -Marc > > ---- > D. Marc Stearman > Lustre Operations Lead > stearman2-i2BcT+NCU+M@public.gmane.org > 925.423.9670 > > > > > On Jul 12, 2013, at 7:34 AM, Chris Hunter <chris.hunter-LrD5EImo2rg@public.gmane.org> wrote: > >> We have a cluster with QDR infiniband. Lustre MDS & OSS servers are >> using mellanox HCA while clients use qlogic HCAs. I believe the switches >> are qlogic QDR 12000 series. We are using ofed distributed with RHEL 6.2. >> >> We currently use the qlogic IB fabric manager (subnet manager). We have >> heard qlogic_fm is good for MPI but bad for lustre. We have the option >> of running ofed opensm. Does anyone have any experience/opinions about >> using qlogic fabric manager versus opensm ? >> >> thanks, >> chris hunter >> yale hpc group >> _______________________________________________ >> Lustre-community mailing list >> Lustre-community-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org >> http://lists.lustre.org/mailman/listinfo/lustre-community