zfs discuss - Oct 2007 - Sequential vs Random I/O on array controller for ZFS usage

We had a Sun Engineer on-site recently who said this:

We should set our array controllers to sequential I/O *even* if we are doing
random I/O if we are using ZFS.   This is because the Arc cache is already
grouping requests up sequentially so to the array controller it will appear like
sequential traffic.

We are using a pair of 3510FC for ZFS pool of Cyrus mail-store data.  Yes lots
of small files and random I/O.  16 gigs RAM so plenty of RAM.  I built 2 RAID-5
LUNs on each array and mirrored a LUN from each array, then pooled the 2 mirror
pairs.  I used largest stripe size on array controller when building the LUNS.

I''m just not convinced argument makes sense or will in fact make any
difference to us.  I''m not even clear what this setting in the array
firmware for sequential versus random DOES other than change the default choice
for stripe size.
 
 
This message posted from opensolaris.org

Le 21 oct. 07 ? 02:40, Vincent Fox a ?crit :
> We had a Sun Engineer on-site recently who said this:
>
> We should set our array controllers to sequential I/O *even* if we  
> are doing random I/O if we are using ZFS.
> This is because the Arc cache is already grouping requests up  
> sequentially so to the array controller it will appear like  
> sequential traffic.
The justification is wrong but the point it true. It''s the copy on  
write design that turns random write into sequential one. Nothing  
related to the ARC cache.
But the point stands.
>
> We are using a pair of 3510FC for ZFS pool of Cyrus mail-store  
> data.  Yes lots of small files and random I/O.  16 gigs RAM so  
> plenty of RAM.  I built 2 RAID-5 LUNs on each array and mirrored a  
> LUN from each array, then pooled the 2 mirror pairs.  I used  
> largest stripe size on array controller when building the LUNS.
>
> I''m just not convinced argument makes sense or will in fact make  
> any difference to us.  I''m not even clear what this setting in the
> array firmware for sequential versus random DOES other than change  
> the default choice for stripe size.
>
Seem ok to me. I would let ZFS do the mirroring for you though(not  
sure what you did). It''s more costly but at least if spurious  
corruption occurs ZFS will
just self-heal that. In the end, ZFS basically expects that 1MB of  
logical volume addresses maps to something physical. No big deal if  
it doesn''t but if it
deviates a lot from this, performance may be different from the  
common installation.
>
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Thanks, we have switched over a couple of our arrays.  Have not noticed a
performance change so perhaps the effect is minor.

Yes we are using ZFS to do the mirroring between the array LUNs and quite happy
with it for reliability.  As someone else said, speed and costs are metrics to
look at but if the data isn''t right nothing else matters.

We build 2 5-drive RAID-5 LUNs in each 3510 and then build 2 mirror pairs and
pool them.  E.g.  zpool create mirror Array1Lun1 Array2Lun1 mirror Array1Lun2
Array2Lun2

I suppose you would call this RAID 5+1+0.

Our hybrid approach of using the array controllers to build LUNs is a workaround
for the ZFS sparing mechanism which is deficient for our needs.  We want to be
able to specify that a spare drive goes only with the drives in it''s
own chassis not for the entire pool and that is how we achieve it.  Thus we are
ensured that no unexpected dependency is introduced into our HA plan when a
drive fails.  There are 2 global spare drives in each array but the array
controllers do the replacement not ZFS.
 
 
This message posted from opensolaris.org

Roch Bourbonnais wrote:
> Le 21 oct. 07 ? 02:40, Vincent Fox a ?crit :
> 
>> We had a Sun Engineer on-site recently who said this:
>>
>> We should set our array controllers to sequential I/O *even* if we  
>> are doing random I/O if we are using ZFS.
>> This is because the Arc cache is already grouping requests up  
>> sequentially so to the array controller it will appear like  
>> sequential traffic.
> 
> The justification is wrong but the point it true. It''s the copy on
> write design that turns random write into sequential one. Nothing  
> related to the ARC cache.
> But the point stands.
I''d say that COW design provides us with opportunity to turn random 
writes into sequential ones, and we try to leverage this opportunity as 
much as possible by allocating new blocks in sequential manner. When 
free space becomes fragmented it''ll be more difficult to allocate
blocks
sequentially so writes may look more random.

victor