Dear All, Whats the maximum write speed can be achieved with Infiniband QDR HCA card. The Question arised from my lustre setup where Iam not able to exceed 500MB/s (write speed) with two OSSs and one MDS Each OSS has four OSTs and one HCA card. I would like to know which is the bottleneck here. I have been using IOR for benchmarking Thank you LakshmaN _________________________________________________________________ New Windows 7: Simplify what you do everyday. Find the right PC for you. http://windows.microsoft.com/shop -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091231/40fbbe44/attachment.html
Hi Lakshmana, You can use standard ofed tools like ib_rdma_bw to check Infiniband bandwidth between pair of nodes. Lustre provides lnet selftest tool for measuring IB bandwidth between pair or group of nodes. Check lustre manual for how tos. Finally, can you post IOR parameters you are using? Cheers, Atul Sent from my iPhone On Dec 31, 2009, at 5:13 PM, lakshmana swamy <klakshman03 at hotmail.com> wrote:> > Dear All, > > Whats the maximum write speed can be achieved with Infiniband QDR > HCA card. > > > The Question arised from my lustre setup where Iam not able to > exceed 500MB/s (write speed) with two OSSs and one MDS > > Each OSS has four OSTs and one HCA card. I would like to know which > is the bottleneck here. > > > I have been using IOR for benchmarking > > Thank you > > LakshmaN > > New Windows 7: Simplify what you do everyday. Find the right PC for > you. > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091231/4ea7f116/attachment.html
ThanQ Atul
I have done the following test, According to the below test, it reaching 1869.74
MB/sec.
This is the command I used for benchmarking.
# mpiexec -machinefile machs1 -np 1024 IOR -N 1024 -a MPIIO -t 1024K -b 1G -F -v
-o /mnt/file1
Max Write: 421.31 MiB/sec (441.77 MB/sec)
Max Read: 1029.68 MiB/sec (1079.70 MB/sec)
***
1. In the aboove command I used MPIIO with 1024 process, I got the almost same
result when I fired same command with 512 process
2. I tested with stripe and without stripe with both the apis MPIIO and POSIX
**
node0:~ # ib_write_bw
------------------------------------------------------------------
RDMA_Write BW Test
Number of qp''s running 1
Connection type : RC
Each Qp will post up to 100 messages each time
Inline data is used up to 1 bytes message
local address: LID 0x100, QPN 0x28004e, PSN 0xc17c83 RKey 0x8002d00 VAddr
0x002b5fd9964000
remote address: LID 0x101, QPN 0x240051, PSN 0x5e61e, RKey 0x8002d00 VAddr
0x002ae797667000
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec]
node0:~ #
node1:~ # ib_write_bw node0
------------------------------------------------------------------
RDMA_Write BW Test
Number of qp''s running 1
Connection type : RC
Each Qp will post up to 100 messages each time
Inline data is used up to 1 bytes message
local address: LID 0x101, QPN 0x240051, PSN 0x5e61e RKey 0x8002d00 VAddr
0x002ae797667000
remote address: LID 0x100, QPN 0x28004e, PSN 0xc17c83, RKey 0x8002d00 VAddr
0x002b5fd9964000
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec]
65536 5000 1869.76 1869.74
------------------------------------------------------------------
node1:~ #
Date: Thu, 31 Dec 2009 17:43:44 +1100
From: Atul.Vidwansa at Sun.COM
Subject: Re: [Lustre-discuss] max write speed of QDR HCA card
To: klakshman03 at hotmail.com
CC: lustre-discuss at lists.lustre.org
Hi Lakshmana,
You can use standard ofed tools like ib_rdma_bw to check Infiniband bandwidth
between pair of nodes.
Lustre provides lnet selftest tool for measuring IB bandwidth between pair or
group of nodes. Check lustre manual for how tos.
Finally, can you post IOR parameters you are using?
Cheers,
Atul
Sent from my iPhone
On Dec 31, 2009, at 5:13 PM, lakshmana swamy <klakshman03 at hotmail.com>
wrote:
Dear All,
Whats the maximum write speed can be achieved with Infiniband QDR HCA card.
The Question arised from my lustre setup where Iam not able to exceed 500MB/s
(write speed) with two OSSs and one MDS
Each OSS has four OSTs and one HCA card. I would like to know which is the
bottleneck here.
I have been using IOR for benchmarking
Thank you
LakshmaN
New Windows 7: Simplify what you do everyday. Find the right PC for you.
_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
_________________________________________________________________
New Windows 7: Find the right PC for you. Learn more.
http://windows.microsoft.com/shop
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091231/53d012b2/attachment.html
Are you sure the IB card is connected at QDR rate?
ibstat will tell you. Look at the "Rate" line.
What kind of machines are you using?
ibstat
CA ''mlx4_0''
CA type: MT26428
Number of ports: 2
Firmware version: 2.6.0
Hardware version: a0
Node GUID: 0x00212800013e5432
System image GUID: 0x00212800013e5435
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 13
LMC: 0
SM lid: 2
Capability mask: 0x02510868
Port GUID: 0x00212800013e5433
Port 2:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 14
LMC: 0
SM lid: 2
Capability mask: 0x02510868
Port GUID: 0x00212800013e5434
On Thu, Dec 31, 2009 at 2:04 AM, lakshmana swamy <klakshman03 at
hotmail.com>wrote:
> ThanQ Atul
>
> I have done the following test, According to the below test, it reaching
> 1869.74 MB/sec.
>
> This is the command I used for benchmarking.
>
> # mpiexec -machinefile machs1 -np 1024 IOR -N 1024 -a MPIIO -t 1024K -b 1G
> -F -v -o /mnt/file1
>
> Max Write: 421.31 MiB/sec (441.77 MB/sec)
> Max Read: 1029.68 MiB/sec (1079.70 MB/sec)
>
> ***
> 1. In the aboove command I used MPIIO with 1024 process, I got the almost
> same result when I fired same command with 512 process
>
> 2. I tested with stripe and without stripe with both the apis MPIIO and
> POSIX
>
>
> **
> node0:~ # ib_write_bw
> ------------------------------------------------------------------
> RDMA_Write BW Test
> Number of qp''s running 1
> Connection type : RC
> Each Qp will post up to 100 messages each time
> Inline data is used up to 1 bytes message
> local address: LID 0x100, QPN 0x28004e, PSN 0xc17c83 RKey 0x8002d00
> VAddr 0x002b5fd9964000
> remote address: LID 0x101, QPN 0x240051, PSN 0x5e61e, RKey 0x8002d00
> VAddr 0x002ae797667000
> Mtu : 2048
> ------------------------------------------------------------------
> #bytes #iterations BW peak[MB/sec] BW average[MB/sec]
> node0:~ #
> node1:~ # ib_write_bw node0
> ------------------------------------------------------------------
> RDMA_Write BW Test
> Number of qp''s running 1
> Connection type : RC
> Each Qp will post up to 100 messages each time
> Inline data is used up to 1 bytes message
> local address: LID 0x101, QPN 0x240051, PSN 0x5e61e RKey 0x8002d00 VAddr
> 0x002ae797667000
> remote address: LID 0x100, QPN 0x28004e, PSN 0xc17c83, RKey 0x8002d00
> VAddr 0x002b5fd9964000
> Mtu : 2048
> ------------------------------------------------------------------
> #bytes #iterations BW peak[MB/sec] BW average[MB/sec]
> 65536 5000 1869.76 1869.74
> ------------------------------------------------------------------
> node1:~ #
>
>
>
>
> ------------------------------
> Date: Thu, 31 Dec 2009 17:43:44 +1100
> From: Atul.Vidwansa at Sun.COM
> Subject: Re: [Lustre-discuss] max write speed of QDR HCA card
> To: klakshman03 at hotmail.com
> CC: lustre-discuss at lists.lustre.org
>
>
> Hi Lakshmana,
>
> You can use standard ofed tools like ib_rdma_bw to check Infiniband
> bandwidth between pair of nodes.
>
> Lustre provides lnet selftest tool for measuring IB bandwidth between pair
> or group of nodes. Check lustre manual for how tos.
>
> Finally, can you post IOR parameters you are using?
>
> Cheers,
> Atul
>
>
>
> Sent from my iPhone
>
> On Dec 31, 2009, at 5:13 PM, lakshmana swamy <klakshman03 at
hotmail.com>
> wrote:
>
>
> Dear All,
>
> Whats the maximum write speed can be achieved with Infiniband QDR HCA
> card.
>
>
> The Question arised from my lustre setup where Iam not able to exceed
> 500MB/s (write speed) with two OSSs and one MDS
>
> Each OSS has four OSTs and one HCA card. I would like to know which is the
> bottleneck here.
>
>
> I have been using IOR for benchmarking
>
> Thank you
>
> LakshmaN
>
> ------------------------------
> New Windows 7: Simplify what you do everyday. Find the right PC for
you.<http://windows.microsoft.com/shop>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
> ------------------------------
> New Windows 7: Find the right PC for you. Learn
more.<http://windows.microsoft.com/shop>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091231/7f19330d/attachment-0001.html
Iam Sorry,
It is DDR not QDR...but still which is the bottleneck
Thank you
laxman
Date: Thu, 31 Dec 2009 12:12:40 -0500
Subject: Re: [Lustre-discuss] max write speed of QDR HCA card
From: erik.froese at gmail.com
To: klakshman03 at hotmail.com
CC: atul.vidwansa at sun.com; lustre-discuss at lists.lustre.org
Are you sure the IB card is connected at QDR rate?
ibstat will tell you. Look at the "Rate" line.
What kind of machines are you using?
ibstat
CA ''mlx4_0''
CA type: MT26428
Number of ports: 2
Firmware version: 2.6.0
Hardware version: a0
Node GUID: 0x00212800013e5432
System image GUID: 0x00212800013e5435
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 13
LMC: 0
SM lid: 2
Capability mask: 0x02510868
Port GUID: 0x00212800013e5433
Port 2:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 14
LMC: 0
SM lid: 2
Capability mask: 0x02510868
Port GUID: 0x00212800013e5434
On Thu, Dec 31, 2009 at 2:04 AM, lakshmana swamy <klakshman03 at
hotmail.com> wrote:
ThanQ Atul
I have done the following test, According to the below test, it reaching 1869.74
MB/sec.
This is the command I used for benchmarking.
# mpiexec -machinefile machs1 -np 1024 IOR -N 1024 -a MPIIO -t 1024K -b 1G -F -v
-o /mnt/file1
Max Write: 421.31 MiB/sec (441.77 MB/sec)
Max Read: 1029.68 MiB/sec (1079.70 MB/sec)
***
1. In the aboove command I used MPIIO with 1024 process, I got the almost same
result when I fired same command with 512 process
2. I tested with stripe and without stripe with both the apis MPIIO and POSIX
**
node0:~ # ib_write_bw
------------------------------------------------------------------
RDMA_Write BW Test
Number of qp''s running 1
Connection type : RC
Each Qp will post up to 100 messages each time
Inline data is used up to 1 bytes message
local address: LID 0x100, QPN 0x28004e, PSN 0xc17c83 RKey 0x8002d00 VAddr
0x002b5fd9964000
remote address: LID 0x101, QPN 0x240051, PSN 0x5e61e, RKey 0x8002d00 VAddr
0x002ae797667000
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec]
node0:~ #
node1:~ # ib_write_bw node0
------------------------------------------------------------------
RDMA_Write BW Test
Number of qp''s running 1
Connection type : RC
Each Qp will post up to 100 messages each time
Inline data is used up to 1 bytes message
local address: LID 0x101, QPN 0x240051, PSN 0x5e61e RKey 0x8002d00 VAddr
0x002ae797667000
remote address: LID 0x100, QPN 0x28004e, PSN 0xc17c83, RKey 0x8002d00 VAddr
0x002b5fd9964000
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec]
65536 5000 1869.76 1869.74
------------------------------------------------------------------
node1:~ #
Date: Thu, 31 Dec 2009 17:43:44 +1100
From: Atul.Vidwansa at Sun.COM
Subject: Re: [Lustre-discuss] max write speed of QDR HCA card
To: klakshman03 at hotmail.com
CC: lustre-discuss at lists.lustre.org
Hi Lakshmana,
You can use standard ofed tools like ib_rdma_bw to check Infiniband bandwidth
between pair of nodes.
Lustre provides lnet selftest tool for measuring IB bandwidth between pair or
group of nodes. Check lustre manual for how tos.
Finally, can you post IOR parameters you are using?
Cheers,
Atul
Sent from my iPhone
On Dec 31, 2009, at 5:13 PM, lakshmana swamy <klakshman03 at hotmail.com>
wrote:
Dear All,
Whats the maximum write speed can be achieved with Infiniband QDR HCA card.
The Question arised from my lustre setup where Iam not able to exceed 500MB/s
(write speed) with two OSSs and one MDS
Each OSS has four OSTs and one HCA card. I would like to know which is the
bottleneck here.
I have been using IOR for benchmarking
Thank you
LakshmaN
New Windows 7: Simplify what you do everyday. Find the right PC for you.
_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
New Windows 7: Find the right PC for you. Learn more.
_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
_________________________________________________________________
Windows 7: Simplify what you do everyday. Find the right PC for you.
http://windows.microsoft.com/shop
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100101/b7460dbd/attachment-0001.html
If you do indeed have DDR infiniband, then your ib_write_bw tests are achieving fairly close to the maximum possible throughput of 2gigabytes/second. As far as where your bottleneck lies, my guess is within the storage connected to your OSSs. What are you using for back- end storage? On Jan 1, 2010, at 1:02 AM, lakshmana swamy wrote:> Iam Sorry, > > It is DDR not QDR...but still which is the bottleneck > > > Thank you > > laxman > > Date: Thu, 31 Dec 2009 12:12:40 -0500 > Subject: Re: [Lustre-discuss] max write speed of QDR HCA card > From: erik.froese at gmail.com > To: klakshman03 at hotmail.com > CC: atul.vidwansa at sun.com; lustre-discuss at lists.lustre.org > > Are you sure the IB card is connected at QDR rate? > > ibstat will tell you. Look at the "Rate" line. > > What kind of machines are you using? > > ibstat > CA ''mlx4_0'' > CA type: MT26428 > Number of ports: 2 > Firmware version: 2.6.0 > Hardware version: a0 > Node GUID: 0x00212800013e5432 > System image GUID: 0x00212800013e5435 > Port 1: > State: Active > Physical state: LinkUp > Rate: 40 > Base lid: 13 > LMC: 0 > SM lid: 2 > Capability mask: 0x02510868 > Port GUID: 0x00212800013e5433 > Port 2: > State: Active > Physical state: LinkUp > Rate: 40 > Base lid: 14 > LMC: 0 > SM lid: 2 > Capability mask: 0x02510868 > Port GUID: 0x00212800013e5434 > > > On Thu, Dec 31, 2009 at 2:04 AM, lakshmana swamy <klakshman03 at hotmail.com > > wrote: > ThanQ Atul > > I have done the following test, According to the below test, it > reaching 1869.74 MB/sec. > > This is the command I used for benchmarking. > > # mpiexec -machinefile machs1 -np 1024 IOR -N 1024 -a MPIIO -t 1024K > -b 1G -F -v -o /mnt/file1 > > Max Write: 421.31 MiB/sec (441.77 MB/sec) > Max Read: 1029.68 MiB/sec (1079.70 MB/sec) > > *** > 1. In the aboove command I used MPIIO with 1024 process, I got the > almost same result when I fired same command with 512 process > > 2. I tested with stripe and without stripe with both the apis MPIIO > and POSIX > > > ** > node0:~ # ib_write_bw > ------------------------------------------------------------------ > RDMA_Write BW Test > Number of qp''s running 1 > Connection type : RC > Each Qp will post up to 100 messages each time > Inline data is used up to 1 bytes message > local address: LID 0x100, QPN 0x28004e, PSN 0xc17c83 RKey > 0x8002d00 VAddr 0x002b5fd9964000 > remote address: LID 0x101, QPN 0x240051, PSN 0x5e61e, RKey > 0x8002d00 VAddr 0x002ae797667000 > Mtu : 2048 > ------------------------------------------------------------------ > #bytes #iterations BW peak[MB/sec] BW average[MB/sec] > node0:~ # > node1:~ # ib_write_bw node0 > ------------------------------------------------------------------ > RDMA_Write BW Test > Number of qp''s running 1 > Connection type : RC > Each Qp will post up to 100 messages each time > Inline data is used up to 1 bytes message > local address: LID 0x101, QPN 0x240051, PSN 0x5e61e RKey > 0x8002d00 VAddr 0x002ae797667000 > remote address: LID 0x100, QPN 0x28004e, PSN 0xc17c83, RKey > 0x8002d00 VAddr 0x002b5fd9964000 > Mtu : 2048 > ------------------------------------------------------------------ > #bytes #iterations BW peak[MB/sec] BW average[MB/sec] > 65536 5000 1869.76 1869.74 > ------------------------------------------------------------------ > node1:~ # > > > > > Date: Thu, 31 Dec 2009 17:43:44 +1100 > From: Atul.Vidwansa at Sun.COM > Subject: Re: [Lustre-discuss] max write speed of QDR HCA card > To: klakshman03 at hotmail.com > CC: lustre-discuss at lists.lustre.org > > > Hi Lakshmana, > > You can use standard ofed tools like ib_rdma_bw to check Infiniband > bandwidth between pair of nodes. > > Lustre provides lnet selftest tool for measuring IB bandwidth > between pair or group of nodes. Check lustre manual for how tos. > > Finally, can you post IOR parameters you are using? > > Cheers, > Atul > > > > Sent from my iPhone > > On Dec 31, 2009, at 5:13 PM, lakshmana swamy > <klakshman03 at hotmail.com> wrote: > > > Dear All, > > Whats the maximum write speed can be achieved with Infiniband QDR > HCA card. > > > The Question arised from my lustre setup where Iam not able to > exceed 500MB/s (write speed) with two OSSs and one MDS > > Each OSS has four OSTs and one HCA card. I would like to know which > is the bottleneck here. > > > I have been using IOR for benchmarking > > Thank you > > LakshmaN > > New Windows 7: Simplify what you do everyday. Find the right PC for > you. > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > New Windows 7: Find the right PC for you. Learn more. > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > > http://windows.microsoft.com/shop Find the right PC for you. > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100101/c960b23e/attachment.html