zfs discuss - Aug 2006 - ZFS & iSCSI: where do do the mirroring/raidz

I have 12 36G disks (in a single D2 enclosure) connected to a V880 that 
I want to "share" to a v40z that is on the same gigabit network
switch.
I''ve already decided that NFS is not the answer - the performance of ON
consolidation builds over NFS just doesn''t cut it for me.

[However think about the problem in the general case - and specifically 
the case of a Thumper as an iSCSI target.]

Both the V880 and the v40z are build machines for the ON consolidation 
so lots of small files reads and writes.

My plan is to use iSCSI to do this given that snv_44 has both the 
initiator and the target parts of iSCSI now.

It seems to make sense that I should be using ZFS on both the target and 
the initiator.

On the initiator the ZFS file system will be the one that users actually 
store data on.

On the target the ZFS file system is just used to gather the disks 
together and present iSCSI targets.

I''m assuming I can''t just "export" the raw disks
over iSCSI - if I can
that might be the better way to do this.

So with that in mind this is my plan so far.

On the target (the V880):
Put all the 12 36G disks into a single zpool (call it iscsitpool).
Use iscsitadm to create 2 targets of 202G each.

On the initiator (the v40z):
Use iscsiadm to discover (import) the 2 202G targets.
Create a zpool that is a mirror of the two 202G targets.

This just doesn''t feel like the best way to do this from an
availability
view point, and I''m not sure at all what to think of the possible
performance at the moment.

So I''m looking for some advice on how best to do this.  I suspect that 
after much discussion some best practice advice might come out of this.



-- 
Darren J Moffat

Darren J Moffat wrote:
> So with that in mind this is my plan so far.
> 
> On the target (the V880):
> Put all the 12 36G disks into a single zpool (call it iscsitpool).
> Use iscsitadm to create 2 targets of 202G each.
> 
> On the initiator (the v40z):
> Use iscsiadm to discover (import) the 2 202G targets.
> Create a zpool that is a mirror of the two 202G targets.
> 
> This just doesn''t feel like the best way to do this from an
availability
> view point, and I''m not sure at all what to think of the possible
> performance at the moment.
 >
> So I''m looking for some advice on how best to do this.  I suspect
that
> after much discussion some best practice advice might come out of this.
Do you want to optimize space, performance, data availability, or data
retention?

Without knowing your answer, I can say that, in general, it takes longer
to recover bigger LUNs/vdevs.  Availability is very dependent on the recovery
time (smaller is better).  To some degree, data retention follows the same
trend.  So, for data availability and retention, you want to use at least one
spare, and smaller, redundant vdevs.  More redundancy really helps retention,
so triple mirror or raidz2 is preferred with regular scrubbing.

This (small vdevs) might also fit well with the dynamic striping for
performance,
but I have no performance data to support my assumption.
  -- richard

Richard Elling wrote:
> Darren J Moffat wrote:
>> So with that in mind this is my plan so far.
>>
>> On the target (the V880):
>> Put all the 12 36G disks into a single zpool (call it iscsitpool).
>> Use iscsitadm to create 2 targets of 202G each.
>>
>> On the initiator (the v40z):
>> Use iscsiadm to discover (import) the 2 202G targets.
>> Create a zpool that is a mirror of the two 202G targets.
>>
>> This just doesn''t feel like the best way to do this from an 
>> availability view point, and I''m not sure at all what to think
of the
>> possible
>> performance at the moment.
>  >
>> So I''m looking for some advice on how best to do this.  I
suspect that
>> after much discussion some best practice advice might come out of this.
> 
> Do you want to optimize space, performance, data availability, or data
> retention?
In order of importance for this use case:

	performance, availability, space, retention.


-- 
Darren J Moffat

On Wed, Darren J Moffat wrote:
> I have 12 36G disks (in a single D2 enclosure) connected to a V880 that 
> I want to "share" to a v40z that is on the same gigabit network
switch.
> I''ve already decided that NFS is not the answer - the performance
of ON
> consolidation builds over NFS just doesn''t cut it for me.
?

With a locally attached 3510 array on a 4-way v40z, I have been 
able to do a full nighly build in 1 hour 7 minutes.  
With NFSv3 access, from the same system, to a couple of 
different NFS servers, I have been able to achieve 1 hour 15 minutes 
in one case and 1 hour 22 minutes in the other.

Is that too slow?

Spencer

Darren J Moffat wrote:
>     performance, availability, space, retention.
OK, something to work with.  I would recommend taking advantage of ZFS''
dynamic stripe over 2-disk mirrors.  This should give good performance,
with good data availability.  If you monitor the status of the disks
regularly, or do not have a 24x7x365 requirement, then you may want the
performance of two more disks over the availability and retention gained
by spares.

In general, the more devices you have, the better performance you can get
(iops * N), but also the worse reliability (MTBF / N).  High availability
is achieved by a combination of reducing risk (diversity), adding
redundancy, and decreasing recovery time (spares).  High retention is
gained by increasing redundancy and decreasing recovery time.

[for the archives]
If you do not have a large up-front performance or space requirement, then
you can take advantage ZFS'' dynamic growth.  For example, if today you
only need 30 GBytes, then you could have a 2-disk mirror with a bunch of
spares.  Spin down (luxadm stop)[1] the spares or turn off the power to the
unused disks (luxadm power_off)[2] to improve their reliability and save power.
As your space needs grow, add disks in mirrored pairs.  This will optimize
your space usage and reliability -> better availability and retention.

[1] somebody will probably chime in and say that this isn''t supported.
It does work well, though.  For spun-down disks, Solaris will start them
when an I/O operation is issued.

[2] may not work for many devices.
  -- richard

Spencer Shepler wrote:
> On Wed, Darren J Moffat wrote:
>> I have 12 36G disks (in a single D2 enclosure) connected to a V880 that
>> I want to "share" to a v40z that is on the same gigabit
network switch.
>> I''ve already decided that NFS is not the answer - the
performance of ON
>> consolidation builds over NFS just doesn''t cut it for me.
> 
> ?
> 
> With a locally attached 3510 array on a 4-way v40z, I have been 
> able to do a full nighly build in 1 hour 7 minutes.  
> With NFSv3 access, from the same system, to a couple of 
> different NFS servers, I have been able to achieve 1 hour 15 minutes 
> in one case and 1 hour 22 minutes in the other.
That would be perfectly acceptable.  I note you do say NFSv3 though and 
not NFSv4.  Is there a reason why you said NFSv3 and not v4 ?  I
haven''t
changed the config on either machine so I''m defaulting to v4.

-- 
Darren J Moffat

On Thu, Darren J Moffat wrote:
> Spencer Shepler wrote:
> >On Wed, Darren J Moffat wrote:
> >>I have 12 36G disks (in a single D2 enclosure) connected to a V880
that
> >>I want to "share" to a v40z that is on the same gigabit
network switch.
> >>I''ve already decided that NFS is not the answer - the
performance of ON
> >>consolidation builds over NFS just doesn''t cut it for me.
> >
> >?
> >
> >With a locally attached 3510 array on a 4-way v40z, I have been 
> >able to do a full nighly build in 1 hour 7 minutes.  
> >With NFSv3 access, from the same system, to a couple of 
> >different NFS servers, I have been able to achieve 1 hour 15 minutes 
> >in one case and 1 hour 22 minutes in the other.
> 
> That would be perfectly acceptable.  I note you do say NFSv3 though and 
> not NFSv4.  Is there a reason why you said NFSv3 and not v4 ?  I
haven''t
> changed the config on either machine so I''m defaulting to v4.
Mainly because that was the data I had at hand.  I have been collecting
various pieces of data and have yet to pick up the NFSv4 data.

There is additional overhead with the NFSv4 client because of the
protocol''s introduction of OPEN/CLOSE operations.  Therefore, for
some workloads and hardware platforms, NFSv4 will be slower.
Builds are one of those thing that is sensitive to hardware platform
at the client.  

Once I get the data, I will followup.

Spencer