zfs discuss - Apr 2007 - concatination & stripe - zfs?

I want to configure my zfs like this :

concatination_stripe_pool :
   concatination
      lun0_controller0
      lun1_controller0
  concatination
      lun2_controller1
      lun3_controller1

1. There is any option to implement it in ZFS?
2. there is other why to get the same configuration?

thanks
 
 
This message posted from opensolaris.org

Hello shay,

Monday, April 23, 2007, 10:14:31 AM, you wrote:

s> I want to configure my zfs like this :

s> concatination_stripe_pool :
s>    concatination
s>       lun0_controller0
s>       lun1_controller0
s>   concatination
s>       lun2_controller1
s>       lun3_controller1

s> 1. There is any option to implement it in ZFS?

  Why not to stripe across all 4?

s> 2. there is other why to get the same configuration?

  SVM+ZFS?

-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

stripe all of them maybe OK.

another 2 questions :
1. there is any concatination mathod in ZFS?

2. I test the prepormance by copy 512MB file into zfs pool that created from 2
luns with stripe,
one      Lun came from Storage-controller-0 and connected through HBA-0
second Lun came from Storage-controller-1 and connected through HBA-1
the time was 12 sec.

And I did another preforman test by copy 512MB file into zfs pool that created
from 1 lun only.
and the test result was the same - 12 sec !?

NOTE : server V240, solaris10(11/06), 2GB RAM, connected to HDS storage type
AMS500 with two HBA type qlogic QLA2342.

Any explanation? it seems that te stripe didn''t actualy effect
preformance!
 
 
This message posted from opensolaris.org

shay wrote:
> stripe all of them maybe OK.
> 
> another 2 questions :
> 1. there is any concatination mathod in ZFS?
No.  ZFS does dynamic striping.  Some will argue that there is no real advantage
to concatenation that does not also exist in dynamic striping.
> 2. I test the prepormance by copy 512MB file into zfs pool that created
from 2 luns with stripe,
> one      Lun came from Storage-controller-0 and connected through HBA-0
> second Lun came from Storage-controller-1 and connected through HBA-1
> the time was 12 sec.
> 
> And I did another preforman test by copy 512MB file into zfs pool that
created from 1 lun only.
> and the test result was the same - 12 sec !?
> 
> NOTE : server V240, solaris10(11/06), 2GB RAM, connected to HDS storage
type AMS500 with two HBA type qlogic QLA2342.
> 
> Any explanation? it seems that te stripe didn''t actualy effect
preformance!
The bottleneck is elsewhere.
  -- richard

shaybery2 at gmail.com said:
> And I did another preforman test by copy 512MB file into zfs pool that
> created from 1 lun only. and the test result was the same - 12 sec !?
> 
> NOTE : server V240, solaris10(11/06), 2GB RAM, connected to HDS storage
type
> AMS500 with two HBA type qlogic QLA2342.
> 
> Any explanation? it seems that te stripe didn''t actualy effect
preformance!
The AMS500 has 1GB cache or more, likely your file isn''t large enough
to
cause any activity on the physical disks.  Also, the ZFS cache gets flushed
out to the array at a time potentially after your I/O test program has
returned and said its writes are complete, so you may not have gotten the
actual time it took to write everything out.

Here''s a simple test that I''ve found captures the time taken
to flush
the ZFS cache to storage (it writes 20GB):

  /bin/time -p /bin/ksh -c "/bin/rm -f testfile && /bin/dd
if=/dev/zero
of=testfile bs=1024k count=20480 && /bin/sync"

Lastly, and you probably are already aware of this, our HDS array turned
out to be quite sensitive to the FC queue length setting.  We got faster
results by setting [s]sd_max_throttle according to the HDS installation
guide for Solaris.  E.g. the guide says 32 is the largest setting you can
use, depending on how many LUN''s are active per port on the array
(Solaris
default value is 256).

Regards,

Marion

- I will try your test.

- But How the zfs cash affect my test? 

- can you send me the guide that tell to change the sd_max_throttle to 32 on
solaris10 or solaris8?
there would me no more then 30 LUN active on the port but no simultanies.

- do you know if a HBA dual QLA2342 can use both 2 ports at the same
time(parallel)? or it uses them one by one?

thanks
 
 
This message posted from opensolaris.org

shaybery2 at gmail.com said:
> - I will try your test.
> - But How the zfs cash affect my test?  
You can measure this yourself.  Try running the test both with and without
the "sync" command at the end.  You should see a faster completion
time
without the "sync", but not all data will have made it to storage. 
How
long the "sync" might take will depend on how much RAM the ZFS ARC is
using, which in turn can depend on how much RAM the system has, among
other things.

> - can you send me the guide that tell to change the sd_max_throttle to 32
on
> solaris10 or solaris8? there would me no more then 30 LUN active on the
port
> but no simultanies. 
I cannot send it to you.  You can find it on the documentation CD that came
with your array;  Your HDS service person can also get it for you.  It should
have "Installation Guide for Solaris" in the document title.  Most of
the
references from Sun on this topic appear to be available only to contract
support customers.  The last one below is freely available:
  http://sunsolve.sun.com/search/document.do?assetkey=1-9-50833-1
  http://sunsolve.sun.com/search/document.do?assetkey=1-9-87146-1
  http://sunsolve.sun.com/search/document.do?assetkey=1-9-27949-1
  http://www.sun.com/bigadmin/jsp/descFile.jsp?url=descAll/solaris_8_performan
> - do you know if a HBA dual QLA2342 can use both 2 ports at the same
> time(parallel)? or it uses them one by one? 
It can use both ports at the same time.

Regards,

Marion

Marion Hakanson wrote:
> shaybery2 at gmail.com said:
>> - I will try your test.
>> - But How the zfs cash affect my test?  
> 
> You can measure this yourself.  Try running the test both with and without
> the "sync" command at the end.  You should see a faster
completion time
> without the "sync", but not all data will have made it to
storage.  How
> long the "sync" might take will depend on how much RAM the ZFS
ARC is
> using, which in turn can depend on how much RAM the system has, among
> other things.
#include standard/sync/disclaimer
DESCRIPTION
      The sync() function writes all information  in  memory  that
      should be on disk, including modified super blocks, modified
      inodes, and delayed block I/O.

      Unlike fsync(3C), which  completes  the  writing  before  it
      returns,  sync() schedules but does not necessarily complete
      the writing before returning.

In other words, the "sync" command schedules a sync.  The consistent
way
to tell if writing is finished is to observe the actual I/O activity.
  -- richard

Richard Elling wrote:
> In other words, the "sync" command schedules a sync.  The
consistent way
> to tell if writing is finished is to observe the actual I/O activity.
ZFS goes beyond this POSIX requirement. When a sync(1M) returns, all 
dirty data that has been cached has been committed to disk.

-Manoj

Those are the result of my preformance test :
time -p "dd if=/dev/zero of=/ZFS_pool/testfile bs=1024k count=20480
&& /bin/sync"
(also I change on the server the kernel parameter -> ssd_max_throttle=32)

seconds__write the file on zpool that biuld from
--------------------------------------------------------
160s        stripeZFS on 2 luns from diffrent HBAs & diffrent
StorageController
190s        stripeZFS on 2 luns from diffrent HBAs & same StorageController
210s        stripeZFS on 2 luns from same dual HBA(diffrent port in the HBA)
&
              diffrent StorageController
210s        ZFS on 1 lun from HBA & StorageController     
300s        local scsi disk on the server 

NOTE : server type V240, solaris10(11/06), 2 internal disks, HBAs QLA2342,mpxio
disable, the server connected to HDS storage type AMS500(2GB cash, Raid 9+1 with
one spare)

Conclutions : 
1. the dual HBA qla2342 is not a parallel at all. so it is better to use 2
single HBAs then one dual HBA.
2. it is better to write Storage from the HBA then to local disks.
 
 
This message posted from opensolaris.org

On 4/25/07, shay <shaybery2 at gmail.com> wrote:
> Those are the result of my preformance test :
>
> Conclutions :
> 1. the dual HBA qla2342 is not a parallel at all. so it is better to use 2
single HBAs then one dual HBA.
That would surprise me. Can it be that you are saturating the PCI slot
you 2342 card
sits in ? IIRC not every slot on V240 can handle dual port 2342 card
going at full rate.
> 2. it is better to write Storage from the HBA then to local disks.
Generalizations like that fairly often become not true at all. It just depends
on too many factors.

-- 
Regards,
        Cyril

>That would surprise me. Can it be that you are saturating the PCI slot
>you 2342 card sits in ? IIRC not every slot on V240 can handle dual port
2342 card
>going at full rate.
I didn''t understand what you mean? 
there is only 3 slots on v240, which of them cannot handle dual HBA?
>Generalizations like that fairly often become not true at all. It just
depends
>on too many factors.
I know that there are many factors for that, but my preformance test was clear ,
it is much faster to write 20GB on SAN than local disk.
 
 
This message posted from opensolaris.org

There are 3 slots in a V240;

1 x 64bit @ 33/66Mhz
2 x 64bit @ 33Mhz

His suggestion was that you might be saturating the PCI slot, since
their respective throughput (in theory) is 528MB and 264MB.

A 2342 should (again, in theory) do 256MB (per port) ... so slotting
the card into the 33Mhz slots won''t help you when you start pushing
data over the second port.

A theory...

On 4/26/07, shay <shaybery2 at gmail.com> wrote:
> >That would surprise me. Can it be that you are saturating the PCI slot
> >you 2342 card sits in ? IIRC not every slot on V240 can handle dual
port 2342 card
> >going at full rate.
>
> I didn''t understand what you mean?
> there is only 3 slots on v240, which of them cannot handle dual HBA?
>
> >Generalizations like that fairly often become not true at all. It just
depends
> >on too many factors.
> I know that there are many factors for that, but my preformance test was
clear , it is much faster to write 20GB on SAN than local disk.
>
>
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

So you think that if I move the QLA2342 from the 33MH slot to the 64MH slot, it
will be faster and will do a real parrallel work on each port?
 
 
This message posted from opensolaris.org

I think I advised you on the solaris forums to try some tests with two separate
ufs filesystems on the luns in question. Just to check that your hardware can
actually do what you think it can.

One other issue that might be confusing things. I believe the AMS is a non
symmetric array so LUN''s have a concept of a native controller. And
will get non optimal performance if you try to access them from the other
controller. Hopefully you have made sure that each lun is native for the
controller your accessing it through.
And hopefully your dealing with LUNS from separate raid groups. So that your
actually trying to write in parallel to two separate sets of spindles. Otherwise
writing in parallel to 2 luns from the same raid group can force the disk heads
to seek back and forth actually significantly decreasing performance.
 
 
This message posted from opensolaris.org