zfs discuss - May 2008 - ZFS pool/filesystem layout design considerations

I''m working with a group who is designing an
application that distributes redundant copies of their data across
multiple server nodes; something akin to RAIS (redundant array of
independent servers).   Within the individual server, they have an
application that stores the particular data into a file on a filesystem
based on a hash or some other means by which to distribute the data
across the various filesystems.

In their early testing, they found performance gains of ZFS compared to
other filesystems.  As they begin to think about the production
implementation they are considering the following design using external
JBOD arrays - each drive is a separate zpool with a single filesystem
defined.  The goal of this design is to reduce the number of
interdependent components and maintain high levels of performance at
the individual node level since they feel their redundancy exists by
having multiple servers with copies of the data.

In recent testing of failure modes, they panic''d the system by removing
a drive and ZFS attempting to protect filesystem integrity (found
references to bug 6322646 filed for more graceful handling of this
condition).  

I have my concerns regarding this design, but I do not
have the in-depth knowledge of ZFS to make the case for or against this
design approach.  I need help to identify the pros/cons so I can
continue the design
discussion?

Thanks,

Todd

-- 
Todd E. Moore
Sun Microsystems Incorporated
443.516.4002
AIM: toddmoore72462


_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On May 14, 2008, at 10:06 AM, Todd E. Moore wrote:
> I''m working with a group who is designing an application that  
> distributes redundant copies of their data across multiple server  
> nodes; something akin to RAIS (redundant array of independent  
> servers).
That part sounds good.
> Within the individual server, they have an application that stores  
> the particular data into a file on a filesystem based on a hash or  
> some other means by which to distribute the data across the various  
> filesystems.
That sounds potentially good, if the underlying filesystems aren''t  
reliable.
>
> In their early testing, they found performance gains of ZFS  
> compared to other filesystems.  As they begin to think about the  
> production implementation they are considering the following design  
> using external JBOD arrays - each drive is a separate zpool with a  
> single filesystem
That seems like a very bad idea to me. If a system has multiple  
drives, using RAIDZ or some equivalent would be much sounder than  
relying on each drive to remain sane. Of course, their multiple  
copies can save them ... unless there''s some correlated event (e.g.  
power surge) that causes failures in multiple drives and even  
multiple systems.
> ...
> I have my concerns regarding this design, but I do not have the in- 
> depth knowledge of ZFS to make the case for or against this design  
> approach.  I need help to identify the pros/cons so I can continue  
> the design discussion?

As you are at Sun, it would seem to me you should tap into the RAS  
expertise and tools available internally to evaluate the  
probabilistic failure modes in the light of field experience with  
various components. I''d expect that multiple systems, each of which  
has RAIDZ ZFS pools (leveraging multiple disks per spool) should have  
much higher RAS figures than the proposed alternative.



-- 
Keith H. Bierman   khbkhb at gmail.com      | AIM kbiermank
5430 Nassau Circle East                  |
Cherry Hills Village, CO 80113           | 303-997-2749
<speaking for myself*> Copyright 2008




-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080514/49a5da93/attachment.html>

Todd E. Moore wrote:
> I''m working with a group who is designing an application that 
> distributes redundant copies of their data across multiple server 
> nodes; something akin to RAIS (redundant array of independent 
> servers).   Within the individual server, they have an application 
> that stores the particular data into a file on a filesystem based on a 
> hash or some other means by which to distribute the data across the 
> various filesystems.
>
> In their early testing, they found performance gains of ZFS compared 
> to other filesystems.  As they begin to think about the production 
> implementation they are considering the following design using 
> external JBOD arrays - each drive is a separate zpool with a single 
> filesystem defined.  The goal of this design is to reduce the number 
> of interdependent components and maintain high levels of performance 
> at the individual node level since they feel their redundancy exists 
> by having multiple servers with copies of the data.
>
> In recent testing of failure modes, they panic''d the system by 
> removing a drive and ZFS attempting to protect filesystem integrity 
> (found references to bug 6322646 filed for more graceful handling of 
> this condition). 
This was fixed in NV b77 along with other FMA enhancements. 
This means that it is in OpenSolaris 2008.05 and should be picked up
for Solaris 10u6.

Note: you can purchase a production support contract for OpenSolaris.
 -- richard
>
> I have my concerns regarding this design, but I do not have the 
> in-depth knowledge of ZFS to make the case for or against this design 
> approach.  I need help to identify the pros/cons so I can continue the 
> design discussion?
>
> Thanks,
>
> Todd
> -- 
> Todd E. Moore
> Sun Microsystems Incorporated
> 443.516.4002
> AIM: toddmoore72462
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>