Hi Guys, Wanted to give a bit more information. So for some reason the transfer rates on my ib interfaces are autonegotiating at 20Gb/s (4X DDR). However, these are QDR HCA''s. Here is the hardware that I have: HP IB 4X QDR PCI-e G2 Dual Port HCA HP 3M 4X DDR/QDR QSFP IB Cu Cables Qlogic 12200 QDR switch I am using all the lustre provided rpms on my servers (RHEL 5.3) and clients (SLES 11). All my servers in this cluster are auto negotiating to 20Gb/s (4X DDR) which should be 40Gb/s (4X QDR). Are any others out there using QDR? If so, did you run into anything similar to this? Is there any specific configuration that is needed for the servers to detect the higher rates. Thanks in advance for your assistance. -J -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100211/a43a2f00/attachment.html
More information:
hpc116:/mnt/SLES11x86_64 # lspci | grep -i mellanox
10:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX IB QDR, PCIe 2.0
5GT/s] (rev a0)
hpc116:/mnt/SLES11x86_64 # ibstatus
Infiniband device ''mlx4_0'' port 1 status:
default gid: fe80:0000:0000:0000:0002:c903:0006:9109
base lid: 0x14
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 20 Gb/sec (4X DDR)
Infiniband device ''mlx4_0'' port 2 status:
default gid: fe80:0000:0000:0000:0002:c903:0006:910a
base lid: 0x0
sm lid: 0x0
state: 1: DOWN
phys state: 2: Polling
rate: 10 Gb/sec (4X)
hpc116:/mnt/SLES11x86_64 # ibstat
CA ''mlx4_0''
CA type: MT26428
Number of ports: 2
Firmware version: 2.6.100
Hardware version: a0
Node GUID: 0x0002c90300069108
System image GUID: 0x0002c9030006910b
Port 1:
State: Active
Physical state: LinkUp
Rate: 20
Base lid: 20
LMC: 0
SM lid: 1
Capability mask: 0x02510868
Port GUID: 0x0002c90300069109
Port 2:
State: Down
Physical state: Polling
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02510868
Port GUID: 0x0002c9030006910a
On Thu, Feb 11, 2010 at 1:21 PM, Jagga Soorma <jagga13 at gmail.com>
wrote:
> Hi Guys,
>
> Wanted to give a bit more information. So for some reason the transfer
> rates on my ib interfaces are autonegotiating at 20Gb/s (4X DDR). However,
> these are QDR HCA''s.
>
> Here is the hardware that I have:
>
> HP IB 4X QDR PCI-e G2 Dual Port HCA
> HP 3M 4X DDR/QDR QSFP IB Cu Cables
> Qlogic 12200 QDR switch
>
> I am using all the lustre provided rpms on my servers (RHEL 5.3) and
> clients (SLES 11). All my servers in this cluster are auto negotiating to
> 20Gb/s (4X DDR) which should be 40Gb/s (4X QDR).
>
> Are any others out there using QDR? If so, did you run into anything
> similar to this? Is there any specific configuration that is needed for
the
> servers to detect the higher rates.
>
> Thanks in advance for your assistance.
>
> -J
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100211/3bf97432/attachment.html
Yet more information. Looks like the switch thinks that this could be set
to 10Gbps (QDR):
hpc116:/mnt/SLES11x86_64 # iblinkinfo.pl -R | grep -i reshpc116
1 34[ ] ==( 4X 5.0 Gbps Active / LinkUp)==> 20 1[ ]
"hpc116 HCA-1" ( Could be 10.0 Gbps)
-J
On Thu, Feb 11, 2010 at 1:26 PM, Jagga Soorma <jagga13 at gmail.com>
wrote:
> More information:
>
> hpc116:/mnt/SLES11x86_64 # lspci | grep -i mellanox
> 10:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX IB QDR, PCIe
> 2.0 5GT/s] (rev a0)
>
> hpc116:/mnt/SLES11x86_64 # ibstatus
> Infiniband device ''mlx4_0'' port 1 status:
> default gid: fe80:0000:0000:0000:0002:c903:0006:9109
> base lid: 0x14
> sm lid: 0x1
> state: 4: ACTIVE
> phys state: 5: LinkUp
> rate: 20 Gb/sec (4X DDR)
>
> Infiniband device ''mlx4_0'' port 2 status:
> default gid: fe80:0000:0000:0000:0002:c903:0006:910a
> base lid: 0x0
> sm lid: 0x0
> state: 1: DOWN
> phys state: 2: Polling
> rate: 10 Gb/sec (4X)
>
> hpc116:/mnt/SLES11x86_64 # ibstat
> CA ''mlx4_0''
> CA type: MT26428
> Number of ports: 2
> Firmware version: 2.6.100
> Hardware version: a0
> Node GUID: 0x0002c90300069108
> System image GUID: 0x0002c9030006910b
> Port 1:
> State: Active
> Physical state: LinkUp
> Rate: 20
> Base lid: 20
> LMC: 0
> SM lid: 1
> Capability mask: 0x02510868
> Port GUID: 0x0002c90300069109
> Port 2:
> State: Down
> Physical state: Polling
> Rate: 10
> Base lid: 0
> LMC: 0
> SM lid: 0
> Capability mask: 0x02510868
> Port GUID: 0x0002c9030006910a
>
>
>
> On Thu, Feb 11, 2010 at 1:21 PM, Jagga Soorma <jagga13 at gmail.com>
wrote:
>
>> Hi Guys,
>>
>> Wanted to give a bit more information. So for some reason the transfer
>> rates on my ib interfaces are autonegotiating at 20Gb/s (4X DDR).
However,
>> these are QDR HCA''s.
>>
>> Here is the hardware that I have:
>>
>> HP IB 4X QDR PCI-e G2 Dual Port HCA
>> HP 3M 4X DDR/QDR QSFP IB Cu Cables
>> Qlogic 12200 QDR switch
>>
>> I am using all the lustre provided rpms on my servers (RHEL 5.3) and
>> clients (SLES 11). All my servers in this cluster are auto negotiating
to
>> 20Gb/s (4X DDR) which should be 40Gb/s (4X QDR).
>>
>> Are any others out there using QDR? If so, did you run into anything
>> similar to this? Is there any specific configuration that is needed
for the
>> servers to detect the higher rates.
>>
>> Thanks in advance for your assistance.
>>
>> -J
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100211/87edceee/attachment.html