zfs discuss - Feb 2007 - Re: ZFS failed Disk Rebuild time on x4500

Nissim Ben Haim wrote:
> I was asked by a customer considering the x4500 - how much time should 
> it take to rebuild a failed Disk under RaidZ ?
> This question keeps popping because customers perceive software RAID as 
> substantially inferior to HW raids.
> I could not find someone who has really measured this under several 
> scenarios.
It is a function of the amount of space used.  As space used -> 0, it
becomes infinitely fast.  As space used -> 100% is approaches the speed
of the I/O subsystem.  In my experience, no hardware RAID array comes
close, they all throttle the resync, though some of them allow you to
tune it a little bit.  The key advantage over a hardware RAID system is
that ZFS knows where the data is and doesn''t need to replicate unused
space.  A hardware RAID array doesn''t know anything about the data, so
it must reconstruct the entire disk.

Also, the reconstruction is done in time order.  See Jeff Bonwick''s
blog:
http://blogs.sun.com/bonwick/entry/smokin_mirrors

I''ve measured resync on some slow IDE disks (*not* an X4500) at an
average
of 20 MBytes/s.  So if you have a 500 GByte drive, that would resync a 100%
full file system in about 7 hours versus 11 days for some other systems
(who shall remain nameless :-)
  -- richard

Hello Richard,

Thursday, February 22, 2007, 3:32:07 AM, you wrote:

RE> Nissim Ben Haim wrote:
>> I was asked by a customer considering the x4500 - how much time should 
>> it take to rebuild a failed Disk under RaidZ ?
>> This question keeps popping because customers perceive software RAID as
>> substantially inferior to HW raids.
>> I could not find someone who has really measured this under several 
>> scenarios.
RE> It is a function of the amount of space used.  As space used -> 0, it
RE> becomes infinitely fast.  As space used -> 100% is approaches the
speed
RE> of the I/O subsystem.  In my experience, no hardware RAID array comes
RE> close, they all throttle the resync, though some of them allow you to
RE> tune it a little bit.  The key advantage over a hardware RAID system is
RE> that ZFS knows where the data is and doesn''t need to replicate
unused
RE> space.  A hardware RAID array doesn''t know anything about the
data, so
RE> it must reconstruct the entire disk.

RE> Also, the reconstruction is done in time order.  See Jeff
Bonwick''s blog:
RE> http://blogs.sun.com/bonwick/entry/smokin_mirrors

RE> I''ve measured resync on some slow IDE disks (*not* an X4500) at
an average
RE> of 20 MBytes/s.  So if you have a 500 GByte drive, that would resync a
100%
RE> full file system in about 7 hours versus 11 days for some other systems
RE> (who shall remain nameless :-)

I wish it worked that good.

raid-z2 made of 11 disks on x4500.
When server wasn''t doing anything - just resync, with pool almost full
with lots of small files, it took about 2 days to re-sync. I think
with lot of small files and almost full pool classic approach to
resync is actually faster.

With a server under some load (not that big) it took about two weeks!
to re-sync disk.

Then you''ve got to remember that you can''t create new
snapshots until
resync finishes or it will start all over again and you never resync.
This is known bug and very, very annoying on x4500 (or any other
server).

You should also keep in mind that current hot-spare support in ZFS is somewhat
lacking - if write fails to a given disk system will panic instead put
hot-spare in. Then after reboot depending on a disk failure type zfs
will note disk is bad and use hot-spare or it will not (if it can
still open that disk).

Also if you have a failing disk (I had) you can manually initiate
re-sync however until it finishes (two weeks???) you can''t replace the
old drive online as it needs to export an pool! You also can''t detach
a disk from raid-z[12] group until you re-sync new one.

IMHO current hot-spare support in ZFS is very basic only and needs lot
of work to be done.

-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

Hello Richard,

Thursday, February 22, 2007, 3:32:07 AM, you wrote:

RE> Nissim Ben Haim wrote:
>> I was asked by a customer considering the x4500 - how much time should 
>> it take to rebuild a failed Disk under RaidZ ?
>> This question keeps popping because customers perceive software RAID as
>> substantially inferior to HW raids.
>> I could not find someone who has really measured this under several 
>> scenarios.
While hot-spare support is currently lacking I can''t agree that SW
RAID is inferior to HW RAID - actually the opposite in many
situations. I see that the customer is considering x4500 - so no
centralized SAN solution is needed.

The other factor is what is his workload and what kind of RAID is he
going to use? With raid-z he/she should keep in mind that random reads
on raid-z won''t give stellar performance, while writes will.



-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

> I''ve measured resync on some slow IDE disks (*not* an X4500) at an
average
> of 20 MBytes/s.  So if you have a 500 GByte drive, that would resync a 100%
> full file system in about 7 hours versus 11 days for some other systems
> 
My experience is that a set of 80% full 250 MB drives took a bit less
than 2 hours each  to replace in a 4x raidz config.  The majority of
space used was taken by large files (isos, music and movie files
(yes, I have teenagers)), although there''s a large number of small
files
as well.  This makes for a performance of a bit less than 40 MB/sec 
during resilvering.  The system was pretty sluggish during this 
operation, but it had only got 1GB of RAM, half of which
firefox wanted :-/.

This was build 55 of Nevada.

- Bart


-- 
Bart Smaalders			Solaris Kernel Performance
barts at cyber.eng.sun.com		http://blogs.sun.com/barts