zfs discuss - May 2008 - Sanity check -- x4500 storage server for enterprise file service

We have been evaluating ZFS as a potential solution for delivering
enterprise file services for our campus. I''ve posted a couple of times
with
various questions, but to recap we want to provide file space to our
approximately 22000 students and 2400 faculty/staff, as well as group
project space for about 1000 groups. Access will be via secure NFSv4 for
our UNIX systems, and CIFS via samba for our windows/macosx clients (the
in-kernel SMB server is not currently an option as we require official
support).

We have almost completed a functional prototype (we''re just waiting for
an
IDR for ACL inheritance so we can complete testing), and are currently
considering deploying x4500 servers. We''re thinking about 5, with
approximately 6000 ZFS filesystems each (Solaris 10U5 still has scalability
issues, any more than about 5-6 thousand filesystems results in
unacceptably long boot cycles).

I was thinking about allocating 2 drives for the OS (SVM mirroring, pending
ZFS boot support), two hot spares, and allocating the other 44 drives as
mirror pairs into a single pool. While this will result in lower available
space than raidz, my understanding is that it should provide much better
performance. Is there anything potentially problematic about this
configuration? Low-level disk performance analysis is not really my field,
I tend to live a bit higher up in the abstraction layer. I don''t think
there would be any performance issues with this, but would much appreciate
commentary from the experts.

Has there been a final resolution on the x4500 I/O hanging issue? I think I
saw a thread the other day about an IDR that seems promising to fix it, if
we go this route hopefully that will be resolved before we go production.

It seems like kind of a waste to allocate 1TB to the operating system,
would there be any issue in taking a slice of those boot disks and creating
a zfs mirror with them to add to the pool?

I''m planning on using snapshots for online backups, maintaining perhaps
10
days worth. At 6000 filesystems, that would be 60000 snapshots floating
around, any potential scalability or performance issues with that?

Any other suggestions or pointing out of potential problems would be
greatly appreciated. So far, ZFS looks like the best available solution
(even better if S10U6 comes out before we go production :) ), thanks to all
of the Sun guys for their great work on that...


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  henson at csupomona.edu
California State Polytechnic University  |  Pomona CA 91768

Paul B. Henson wrote:
> We have been evaluating ZFS as a potential solution for delivering
> enterprise file services for our campus. I''ve posted a couple of
times with
> various questions, but to recap we want to provide file space to our
> approximately 22000 students and 2400 faculty/staff, as well as group
> project space for about 1000 groups. Access will be via secure NFSv4 for
> our UNIX systems, and CIFS via samba for our windows/macosx clients (the
> in-kernel SMB server is not currently an option as we require official
> support).
>   
N.B. anyone can purchase a Production Subscription for OpenSolaris
which would get both "support" and the in-kernel CIFS server.
http://www.sun.com/service/opensolaris/index.jsp

<sidebar>
At USC, we have a deal with Google whereby we use Google apps
and gmail, so if you send e-mail to me @usc.edu, then I get it as a
gmail service.  The interesting bit is that it uses USC''s single sign
on
infrastructure, not Google''s.
</sidebar>
> We have almost completed a functional prototype (we''re just
waiting for an
> IDR for ACL inheritance so we can complete testing), and are currently
> considering deploying x4500 servers. We''re thinking about 5, with
> approximately 6000 ZFS filesystems each (Solaris 10U5 still has scalability
> issues, any more than about 5-6 thousand filesystems results in
> unacceptably long boot cycles).
>
> I was thinking about allocating 2 drives for the OS (SVM mirroring, pending
> ZFS boot support), two hot spares, and allocating the other 44 drives as
> mirror pairs into a single pool. While this will result in lower available
> space than raidz, my understanding is that it should provide much better
> performance. Is there anything potentially problematic about this
> configuration? Low-level disk performance analysis is not really my field,
> I tend to live a bit higher up in the abstraction layer. I don''t
think
> there would be any performance issues with this, but would much appreciate
> commentary from the experts.
>   
That is what I would do.
> Has there been a final resolution on the x4500 I/O hanging issue? I think I
> saw a thread the other day about an IDR that seems promising to fix it, if
> we go this route hopefully that will be resolved before we go production.
>
> It seems like kind of a waste to allocate 1TB to the operating system,
> would there be any issue in taking a slice of those boot disks and creating
> a zfs mirror with them to add to the pool?
>   
This is also what I would do.
> I''m planning on using snapshots for online backups, maintaining
perhaps 10
> days worth. At 6000 filesystems, that would be 60000 snapshots floating
> around, any potential scalability or performance issues with that?
>   
I don''t think we have much data for this size of a production system.
OTOH, I would expect that only a small subset of the space will be
active.
> Any other suggestions or pointing out of potential problems would be
> greatly appreciated. So far, ZFS looks like the best available solution
> (even better if S10U6 comes out before we go production :) ), thanks to all
> of the Sun guys for their great work on that...
>
>   
Long-term backup is more difficult.  Is there an SLA, or do you need to
treat faculty/staff different from undergrads or grad students?
 -- richard

On Wed, 7 May 2008, Paul B. Henson wrote:
>
> I was thinking about allocating 2 drives for the OS (SVM mirroring, pending
> ZFS boot support), two hot spares, and allocating the other 44 drives as
> mirror pairs into a single pool. While this will result in lower available
> space than raidz, my understanding is that it should provide much better
> performance. Is there anything potentially problematic about this
> configuration? Low-level disk performance analysis is not really my field,
It sounds quite solid.  The load should be quite nicely distributed 
across the mirrors.
> It seems like kind of a waste to allocate 1TB to the operating system,
> would there be any issue in taking a slice of those boot disks and creating
> a zfs mirror with them to add to the pool?
You don''t want to go there.  Keep in mind that there is currently no 
way to reclaim a device after it has been added to the pool other than 
substituting another device for it.  Also, the write performance to 
these slices would be less than normal.

If I was you, I would keep more disks spare in the beginning and see 
how the system is working.  If everything is working great, then add 
more disks to the pool.  Once disks are added to the pool, they are 
comitted.

An advantage of load-shared mirrors is that more pairs can be added at 
any time.  You need enough disks in the system to satisfy current disk 
space and I/O rate requirements, but it is not necessary to start off 
with all the disks added to the pool.  Disks added earlier will be 
initially more loaded up than disks added later.

Bob
=====================================Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Wed, May 7, 2008 at 11:34 PM, Paul B. Henson <henson at acm.org>
wrote:
>
> We have been evaluating ZFS as a potential solution for delivering
> enterprise file services for our campus.
...
> I was thinking about allocating 2 drives for the OS (SVM mirroring, pending
> ZFS boot support), two hot spares, and allocating the other 44 drives as
> mirror pairs into a single pool. While this will result in lower available
> space than raidz, my understanding is that it should provide much better
> performance.
As a regular fileserver, yes - random reads of small files on raidz
isn''t
too hot...
> Has there been a final resolution on the x4500 I/O hanging issue? I think I
> saw a thread the other day about an IDR that seems promising to fix it, if
> we go this route hopefully that will be resolved before we go production.
I just disable NCQ and have done with it.
> It seems like kind of a waste to allocate 1TB to the operating system,
> would there be any issue in taking a slice of those boot disks and creating
> a zfs mirror with them to add to the pool?
Personally, I wouldn''t - I do like pool-level separation of data and
OS.

What I normally do in these cases is to create a separate pool
and use it for something else useful.
> I''m planning on using snapshots for online backups, maintaining
perhaps 10
> days worth. At 6000 filesystems, that would be 60000 snapshots floating
> around, any potential scalability or performance issues with that?
My only concern here would be how hard it would be to delete the
snapshots. With that cycle, you''re deleting 6000 snapshots a day,
and while snapshot creation is "free", my experience is that snapshot
deletion is not.

-- 
-Peter Tribble
http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/

Mirrored drives should be fine.  My understanding is that write performance
suffers slightly in a mirrored configuration, but random reads are much faster. 
In your scenario I would expect mirroring to give far superior performance than
raid-z2.

We''re looking to do something similar, but we''re strongly
considering dual parity mirrors for when we buy some Thumpers.  You''re
getting tons of storage for your money with these servers, but the rebuild time
and risk of data loss is considerable when you''re dealing with busy
500GB drives.  We want to ensure that we never have to try to restore a 24TB
Thumper from tape backup, and that our data is protected even if a disk fails.

I found this post quite an interesting read:
http://blogs.sun.com/relling/entry/raid_recommendations_space_vs_mttdl

It may be an obvious point, but are you aware that snapshots need to be stopped
any time a disk fails?  It''s something to consider if you''re
planning frequent snapshots.

Regarding the OS, I wouldn''t even attempt to use those disks for data. 
When we buy x4500''s I''ll be buying a couple of spare 500GB
disks so I can mirror the boot volume onto them and stick them on the shelf,
just in case.  A 500GB disk costs ?60 or so, is it really worth risking your
server over it?

And finally, I''ve no idea what performance would be like with that many
snapshots, but Sun do a 60 day free trial of that server, so if you
haven''t done so already, take advantage of that and test it for
yourself.
 
 
This message posted from opensolaris.org

On Thu, 8 May 2008, Ross wrote:
> protected even if a disk fails. I found this post quite an 
> interesting 
> read:http://blogs.sun.com/relling/entry/raid_recommendations_space_vs_mttdl
Richard''s blog entry does not tell the whole story.  ZFS does not 
protect against memory corruption errors and CPU execution errors 
except for in the validated data path.  It also does not protect you 
against kernel bugs, corrosion, meteorite strikes, or civil unrest. 
As a result, the MTTDL plots (which only consider media reliability 
and redundancy) become quite incorrect as they reach stratospheric 
levels.

Note that Richard does include a critical disclaimer: "The MTTDL 
calculation is one attribute of Reliability, Availability, and 
Serviceability (RAS) which we can also calculate relatively easily." 
Notice the operative word "one".

The law of diminishing returns still applies.

Bob
=====================================Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Thu, 8 May 2008, Ross Smith wrote:
> True, but I''m seeing more and more articles pointing out that the 
> risk of a secondary failure is increasing as disks grow in size, and
Quite true.
> While I''m not sure of the actual error rates (Western digital list
> their unrecoverable rates as < 1 in 10^15), I''m very concious
that
> if you have any one disk fail completely, you are then reliant on 
> being able to read without error every single bit of data from every 
> other disk in that raid set.  I''d much rather have dual parity and
> know that single bit errors are still easily recoverable during the 
> rebuild process.
I understand the concern.  However, the published unrecoverable rates 
are for the completely random write/read case.  ZFS validates the data 
read for each read and performs a repair if a read is faulty.  Doing a 
"zfs scrub" forces all of the data to be read and repaired if 
necessary.  Assuming that the data is read (and repaired if necessary) 
on a periodic basis, the chance that an unrecoverable read will occur 
will surely be dramatically lower.  This of course assumes that the 
system administrator pays attention and proactively replaces disks 
which are reporting unusually high and increasing read failure rates.

It is a simple matter of statistics.  If you have read a disk block 
successfully 1000 times, what is the probability that the next read 
from that block will spontaneously fail?  How about if you have read 
from it successfully a million times?

Assuming a reasonably designed storage system, the most likely cause 
of data loss is human error due to carelessness or confusion.

Bob
=====================================Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On 05/08/2008 08:11 AM, Ross wrote:
> It may be an obvious point, but are you aware that snapshots need to be
stopped any time a disk fails?  It''s something to consider if
you''re planning frequent snapshots.
I''ve never heard this before. Why would snapshots need to be stopped
for
a disk failure?

--
Dave

Dave wrote:
> On 05/08/2008 08:11 AM, Ross wrote:
>   
>> It may be an obvious point, but are you aware that snapshots need to be
stopped any time a disk fails?  It''s something to consider if
you''re planning frequent snapshots.
>>     
>
> I''ve never heard this before. Why would snapshots need to be
stopped for
> a disk failure?
>   
Because taking a snapshot makes the scrub start over.  I hadn''t thought
about this extending to a resilver, but I guess it would!

Anyway, I take frequent snapshots on my home ZFS server, and I got tired 
of a 90 minute scrub that started over every 60 minutes.  So, I put the 
following code snipped into my snapshot-management script:

    # Is a scrub in-progress?  If so, abort
    if [ ! $(zpool status | grep -c ''scrub in progress'' ) == 0
]
    then
            exit -1;
    fi
      

Now I skip a couple of snashots a day, but I can run a daily scrub to 
make sure that my photos and the code from my undergraduate CS projects 
are being coherently stored.

To solve the resilver problem, change the "grep" statement to
something
like "egrep -c ''scrub in progress|resilver in progress''
".

-Luke

Bob Friesenhahn wrote:
> On Thu, 8 May 2008, Ross wrote:
>
>   
>> protected even if a disk fails. I found this post quite an 
>> interesting 
>>
read:http://blogs.sun.com/relling/entry/raid_recommendations_space_vs_mttdl
>>     
>
> Richard''s blog entry does not tell the whole story.  ZFS does not 
> protect against memory corruption errors and CPU execution errors 
> except for in the validated data path.  It also does not protect you 
> against kernel bugs, corrosion, meteorite strikes, or civil unrest. 
> As a result, the MTTDL plots (which only consider media reliability 
> and redundancy) become quite incorrect as they reach stratospheric 
> levels.
>   
These are statistical models, or as they say, "every child in Lake 
Woebegon is
above average." :-)  The important take-away is that no protection 
sucks, single
parity protection is better, and double parity protection is even 
better. See also
the discussion on "mean time" measurements and when we don''t
like them at
http://blogs.sun.com/relling/entry/using_mtbf_and_time_dependent
 -- richard
> Note that Richard does include a critical disclaimer: "The MTTDL 
> calculation is one attribute of Reliability, Availability, and 
> Serviceability (RAS) which we can also calculate relatively easily." 
> Notice the operative word "one".
>
> The law of diminishing returns still applies.
>
> Bob
> =====================================> Bob Friesenhahn
> bfriesen at simple.dallas.tx.us,
http://www.simplesystems.org/users/bfriesen/
> GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

Luke Scharf wrote:
> Dave wrote:
>> On 05/08/2008 08:11 AM, Ross wrote:
>>   
>>> It may be an obvious point, but are you aware that snapshots need
to be stopped any time a disk fails?  It''s something to consider if
you''re planning frequent snapshots.
>>>     
>> I''ve never heard this before. Why would snapshots need to be
stopped for
>> a disk failure?
>>   
> 
> Because taking a snapshot makes the scrub start over.  I hadn''t
thought
> about this extending to a resilver, but I guess it would!
I thought this was fixed in OpenSolaris and Solaris 10 U5? Can one of 
the ZFS folks please comment?

I''ll probably get to test this the hard way next week, as I start to 
attempt to engineer a zfs send/recv DR solution. If this bug _isn''t_ 
fixed, I will be a very unhappy geek :-(

-- 
Carson

zfs-discuss-bounces at opensolaris.org wrote on 05/08/2008 02:31:43 PM:
> Luke Scharf wrote:
> > Dave wrote:
> >> On 05/08/2008 08:11 AM, Ross wrote:
> >>
> >>> It may be an obvious point, but are you aware that snapshots
> need to be stopped any time a disk fails?  It''s something to
> consider if you''re planning frequent snapshots.
> >>>
> >> I''ve never heard this before. Why would snapshots need to
be stopped
for
> >> a disk failure?
> >>
> >
> > Because taking a snapshot makes the scrub start over.  I
hadn''t thought
> > about this extending to a resilver, but I guess it would!
>
> I thought this was fixed in OpenSolaris and Solaris 10 U5? Can one of
> the ZFS folks please comment?
>
> I''ll probably get to test this the hard way next week, as I start
to
> attempt to engineer a zfs send/recv DR solution. If this bug
_isn''t_
> fixed, I will be a very unhappy geek :-(
>
Sorry to hear you are unhappy. =(

It is not fixed yet -- I am actively looking for the fix too.  On a 4500
with a lot of used data on a large pool,  you can expect to lose snapshots
for 5+ days to resilver or scrub.

On May 8, 2008, at 12:31 PM, Carson Gaspar wrote:
> Luke Scharf wrote:
>> Dave wrote:
>>> On 05/08/2008 08:11 AM, Ross wrote:
>>>
>>>> It may be an obvious point, but are you aware that snapshots
need
>>>> to be stopped any time a disk fails?  It''s something
to consider
>>>> if you''re planning frequent snapshots.
>>>>
>>> I''ve never heard this before. Why would snapshots need to
be
>>> stopped for
>>> a disk failure?
>>>
>>
>> Because taking a snapshot makes the scrub start over.  I
hadn''t
>> thought
>> about this extending to a resilver, but I guess it would!
>
> I thought this was fixed in OpenSolaris and Solaris 10 U5? Can one of
> the ZFS folks please comment?
Matt just sent out a code review for this today:
6343667 scrub/resilver has to start over when a snapshot is taken
http://bugs.opensolaris.org/view_bug.do?bug_id=6343667

eric

On 05/08/2008 11:29 AM, Luke Scharf wrote:
> Dave wrote:
>> On 05/08/2008 08:11 AM, Ross wrote:
>>  
>>> It may be an obvious point, but are you aware that snapshots need
to
>>> be stopped any time a disk fails?  It''s something to
consider if
>>> you''re planning frequent snapshots.
>>>     
>>
>> I''ve never heard this before. Why would snapshots need to be
stopped
>> for a disk failure?
>>   
> 
> Because taking a snapshot makes the scrub start over.  I hadn''t
thought
> about this extending to a resilver, but I guess it would!
> 
Ah, yes, for scrubs/resilvers. My brain didn''t seem to understand the 
actual intent of Ross'' statement, which was to say that repairing a 
mirror/raidz after replacing a bad disk requires halting new snapshots. 
On the other hand, a disk can fail and you can take snapshots all day 
long on a degraded pool.

Glad to hear there''s code under review to fix this.

--
Dave

On Wed, 7 May 2008, Richard Elling wrote:
> N.B. anyone can purchase a Production Subscription for OpenSolaris which
> would get both "support" and the in-kernel CIFS server.
> http://www.sun.com/service/opensolaris/index.jsp
Wow. That''s new, and very intriguing. Any idea on the potential
timeline
for support of Windows shadow copy in the in-kernel CIFS server? That''s
one
feature offered by Samba which I understand is an RFE for the in-kernel
CIFS server.

> Long-term backup is more difficult.  Is there an SLA, or do you need to
> treat faculty/staff different from undergrads or grad students?
I don''t know that we have an official SLA, for the most part in the
context
of storage we treat everyone the same other than quota allocation.


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  henson at csupomona.edu
California State Polytechnic University  |  Pomona CA 91768

On Wed, 7 May 2008, Bob Friesenhahn wrote:
> > It seems like kind of a waste to allocate 1TB to the operating system,
> > would there be any issue in taking a slice of those boot disks and
> > creating a zfs mirror with them to add to the pool?
>
> You don''t want to go there.  Keep in mind that there is currently
no way
> to reclaim a device after it has been added to the pool other than
> substituting another device for it.  Also, the write performance to these
> slices would be less than normal.
That''s a good point; I do recall reading that ZFS is more efficient
when
given an entire disk rather than a slice.

Thanks for the input...


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  henson at csupomona.edu
California State Polytechnic University  |  Pomona CA 91768

On Thu, 8 May 2008, Peter Tribble wrote:
> As a regular fileserver, yes - random reads of small files on raidz
isn''t
> too hot...
There that would pretty much be our usage scenario; home directories and
group project directories.
> I just disable NCQ and have done with it.
Doesn''t that result in a performance decrease? (I suppose not as much
as
the hanging IO issue, but still less than ideal?).
> What I normally do in these cases is to create a separate pool
> and use it for something else useful.
I''m not sure what else I would do with almost a terabyte of storage,
the
management overhead of dealing with it outside of the other storage would
perhaps cost more than just ignoring it.
> My only concern here would be how hard it would be to delete the
> snapshots. With that cycle, you''re deleting 6000 snapshots a day,
and
> while snapshot creation is "free", my experience is that snapshot
> deletion is not.
I did some testing with up to 10,000 filesystems,  but with minimal
activity between snapshots will probably wasn''t a valid usage scenario
to
verify the resource requirements.

Thanks...


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  henson at csupomona.edu
California State Polytechnic University  |  Pomona CA 91768

On Thu, 8 May 2008, eric kustarz wrote:
> Matt just sent out a code review for this today:
> 6343667 scrub/resilver has to start over when a snapshot is taken
> http://bugs.opensolaris.org/view_bug.do?bug_id=6343667
Wow, this bug was originally opened 30-OCT-2005... I guess it was really
difficult to fix? From a functionality perspective, it seems pretty
detrimental.

-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  henson at csupomona.edu
California State Polytechnic University  |  Pomona CA 91768

Yeah, it''s a *very* old bug.  The main reason we put our ZFS rollout on
hold was concerns over reliability with such an old (and imo critical) bug still
present in the system.
 
 
This message posted from opensolaris.org

Hi all,
until the scrub problem (http://bugs.opensolaris.org/view_bug.do?bug_id=6343667 
) is fixed,you should be able to "simulate" a scrub on the latest data
by using
zfs send <snapshot> > /dev/null
Since the primary purpose is to verify latent bugs and to have zfs  
auto-correct them, simply reading all data would be sufficient to  
achieve the same purpose.
Problems:
1. This does not verify data from older snapshots and has to be issued  
for each FS in the pool.
2. It might be hard to schedule this task as comfortable as a scrub.

Resilvering should pose less of a problem as that only has to rewrite  
data of a single disk, i.e. you do not have to stop snapshots for a  
very long time. If a device was only temporarily unavailable,  
resilvering is actually much faster as only affected blocks will be re- 
written.

On Mon, May 12, 2008 at 06:44:39PM +0200, Ralf Bertling
wrote:
> ...you should be able to "simulate" a scrub on the latest data by
> using
> zfs send <snapshot> > /dev/null
> Since the primary purpose is to verify latent bugs and to have zfs  
> auto-correct them, simply reading all data would be sufficient to  
> achieve the same purpose.
> Problems:
> 1. This does not verify data from older snapshots and has to be issued  
> for each FS in the pool.
> 2. It might be hard to schedule this task as comfortable as a scrub.
It also won''t check redundant copies or parity data.  Does a scrub do
that?

-- 
Darren Dunham                                           ddunham at taos.com
Senior Technical Consultant         TAOS            http://www.taos.com/
Got some Dr Pepper?                           San Francisco, CA bay area
         < This line left intentionally blank to confuse you. >