Lustre discuss - Apr 2009 - Clarification on DDN performance best practices

The Lustre manual (DDN tuning section) suggest the following:

 > On the S2A 9500 and 9550, we measured significantly better 
performance with 2 tiers per LUN

I''ve seen conflicting statements regarding this. Are you saying that 2 
tiers should be the max? Or is it advantageous to create OSTs that span 
many more? I''ve seen indication of up to 12 tiers to maximize write 
performance. Any insight would help.

Also, when creating LUNs on a DDN to be use exclusively for OSTs, does 
in make sense to use a disk block size of 4K (instead of 512bytes) since 
the standard block size for luster is also 4K.

Thanks,
/pgc

> I''ve seen conflicting statements regarding this. Are you saying
that 2
> tiers should be the max? Or is it advantageous to create OSTs that span 
> many more? I''ve seen indication of up to 12 tiers to maximize
write
> performance. Any insight would help.
>
>   
If I recall correctly, the test was performed with 1 tier per LUN vs two 
tiers per LUN, and two tiers had better performance. However, with the 
larger drives, especially in the SATA world, you quickly start to hit 
the 8TB ext3 limit. With the 1TB drives, you hit that limit in one tier, 
so obviously this advice is not applicable to these large
disks.
> Also, when creating LUNs on a DDN to be use exclusively for OSTs, does 
> in make sense to use a disk block size of 4K (instead of 512bytes) since 
> the standard block size for luster is also 4K.
>   
4k blocks are also a good idea because the S2As prefer that IO be 
aligned on 4k boundries.

- Kit
> Thanks,
> /pgc
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>   

-- 
---
Kit Westneat
kwestneat at datadirectnet.com
812-484-8485

[ ... ]
>> Also, when creating LUNs on a DDN to be use exclusively for OSTs,
>> does in make sense to use a disk block size of 4K (instead of
>> 512bytes) since the standard block size for luster is also 4K.
> 4k blocks are also a good idea because the S2As prefer that IO be
> aligned on 4k boundries.
As to this, the tier are organized as something similar to 8+2 RAID6.
For 8-way data, which is the unit? Sectors? Blocks? In other words, what
is the stripe data size in KiB with a sector size of 512 vs. a sector
size of 4KiB?

> As to this, the tier are organized as something similar to 8+2 RAID6.
> For 8-way data, which is the unit? Sectors? Blocks? In other words, what
> is the stripe data size in KiB with a sector size of 512 vs. a sector
> size of 4KiB?
>   
The S2A directRAID6 is actually based on RAID3, so we use byte striping 
instead of block striping. The 4k alignment comes from the physical disk 
sector size of 512-bytes (512 * 8 = 4k). The S2A has a cache segment 
size, which while independent of the LUN sector size, is sort of similar 
in some ways to a block size on a RAID5, though it will never do a 
read-modify-write. The cache size can be set between 128k - 2M, and for 
Lustre, we usually set it to 1 or 2M, since Lustre currently operates 
with 1M messages.

This picture sort of describes from a high level what happens on the 
data path (the text is very marketing heavy unfortunately):
http://ddn.com/index.php?id=48

If you are going to be at LUG, DDN will have a fair number folks 
including myself there, feel free to come say hi. We''ll also be talking
at 1 on Friday.

HTH,
Kit

-- 
---
Kit Westneat
kwestneat at datadirectnet.com
812-484-8485

What kernel does DDN use for its products? Or is that a closed secret?


On Mon, Apr 13, 2009 at 10:17 AM, Kit Westneat <kwestneat at ddn.com>
wrote:
>
>> As to this, the tier are organized as something similar to 8+2 RAID6.
>> For 8-way data, which is the unit? Sectors? Blocks? In other words,
what
>> is the stripe data size in KiB with a sector size of 512 vs. a sector
>> size of 4KiB?
>>
> The S2A directRAID6 is actually based on RAID3, so we use byte striping
> instead of block striping. The 4k alignment comes from the physical disk
> sector size of 512-bytes (512 * 8 = 4k). The S2A has a cache segment
> size, which while independent of the LUN sector size, is sort of similar
> in some ways to a block size on a RAID5, though it will never do a
> read-modify-write. The cache size can be set between 128k - 2M, and for
> Lustre, we usually set it to 1 or 2M, since Lustre currently operates
> with 1M messages.
>
> This picture sort of describes from a high level what happens on the
> data path (the text is very marketing heavy unfortunately):
> http://ddn.com/index.php?id=48
>
> If you are going to be at LUG, DDN will have a fair number folks
> including myself there, feel free to come say hi. We''ll also be
talking
> at 1 on Friday.
>
> HTH,
> Kit
>
> --
> ---
> Kit Westneat
> kwestneat at datadirectnet.com
> 812-484-8485
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>

Our storage systems just present as scsi devices either using fc or ib srp. When
we do Lustre installations, we just use whatever is most stable. Most recently
we have been doing a lot of rhel5 based installs, so just use the sun provided
rhel5 lustre kernel.

- Kit

----- Original Message -----
From: Mag Gam <magawake at gmail.com>
To: Kit Westneat
Cc: Peter Grandi <pg_lus at lus.for.sabi.co.uk>; List Lustre discussion
<lustre-discuss at lists.lustre.org>
Sent: Wed Apr 15 20:11:41 2009
Subject: Re: [Lustre-discuss] Clarification on DDN performance best practices

What kernel does DDN use for its products? Or is that a closed secret?


On Mon, Apr 13, 2009 at 10:17 AM, Kit Westneat <kwestneat at ddn.com>
wrote:
>
>> As to this, the tier are organized as something similar to 8+2 RAID6.
>> For 8-way data, which is the unit? Sectors? Blocks? In other words,
what
>> is the stripe data size in KiB with a sector size of 512 vs. a sector
>> size of 4KiB?
>>
> The S2A directRAID6 is actually based on RAID3, so we use byte striping
> instead of block striping. The 4k alignment comes from the physical disk
> sector size of 512-bytes (512 * 8 = 4k). The S2A has a cache segment
> size, which while independent of the LUN sector size, is sort of similar
> in some ways to a block size on a RAID5, though it will never do a
> read-modify-write. The cache size can be set between 128k - 2M, and for
> Lustre, we usually set it to 1 or 2M, since Lustre currently operates
> with 1M messages.
>
> This picture sort of describes from a high level what happens on the
> data path (the text is very marketing heavy unfortunately):
> http://ddn.com/index.php?id=48
>
> If you are going to be at LUG, DDN will have a fair number folks
> including myself there, feel free to come say hi. We''ll also be
talking
> at 1 on Friday.
>
> HTH,
> Kit
>
> --
> ---
> Kit Westneat
> kwestneat at datadirectnet.com
> 812-484-8485
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>

Kit:

Thanks for the reply. I was under the impression you create your own
version of Lustre and then sell it as a "blackbox" to the customers.

I think Terascala does this, and I would like to figure out exactly
what they include in their Lustre offering.

TIA

On Wed, Apr 15, 2009 at 11:37 PM, Kit Westneat <kwestneat at ddn.com>
wrote:
> Our storage systems just present as scsi devices either using fc or ib srp.
When we do Lustre installations, we just use whatever is most stable. Most
recently we have been doing a lot of rhel5 based installs, so just use the sun
provided rhel5 lustre kernel.
>
> - Kit
>
> ----- Original Message -----
> From: Mag Gam <magawake at gmail.com>
> To: Kit Westneat
> Cc: Peter Grandi <pg_lus at lus.for.sabi.co.uk>; List Lustre
discussion <lustre-discuss at lists.lustre.org>
> Sent: Wed Apr 15 20:11:41 2009
> Subject: Re: [Lustre-discuss] Clarification on DDN performance best
practices
>
> What kernel does DDN use for its products? Or is that a closed secret?
>
>
> On Mon, Apr 13, 2009 at 10:17 AM, Kit Westneat <kwestneat at ddn.com>
wrote:
>>
>>> As to this, the tier are organized as something similar to 8+2
RAID6.
>>> For 8-way data, which is the unit? Sectors? Blocks? In other words,
what
>>> is the stripe data size in KiB with a sector size of 512 vs. a
sector
>>> size of 4KiB?
>>>
>> The S2A directRAID6 is actually based on RAID3, so we use byte striping
>> instead of block striping. The 4k alignment comes from the physical
disk
>> sector size of 512-bytes (512 * 8 = 4k). The S2A has a cache segment
>> size, which while independent of the LUN sector size, is sort of
similar
>> in some ways to a block size on a RAID5, though it will never do a
>> read-modify-write. The cache size can be set between 128k - 2M, and for
>> Lustre, we usually set it to 1 or 2M, since Lustre currently operates
>> with 1M messages.
>>
>> This picture sort of describes from a high level what happens on the
>> data path (the text is very marketing heavy unfortunately):
>> http://ddn.com/index.php?id=48
>>
>> If you are going to be at LUG, DDN will have a fair number folks
>> including myself there, feel free to come say hi. We''ll also
be talking
>> at 1 on Friday.
>>
>> HTH,
>> Kit
>>
>> --
>> ---
>> Kit Westneat
>> kwestneat at datadirectnet.com
>> 812-484-8485
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>