Lustre discuss - Jun 2011 - ZFS question: HW raid5 vs raidz?

I was checking out zfsonlinux.org to see how things have been going
lately and I had a question. Whats the difference, or whats better:
Use a hardware raid5(or 6) or use zfs to create a raidz pool? In terms
of Lustre, is one preferred over another?

David

-- 
Personally, I liked the university. They gave us money and facilities,
we didn''t have to produce anything! You''ve never been out of
college!
You don''t know what it''s like out there! I''ve worked
in the private
sector. They expect results. -Ray Ghostbusters

On 2011-06-10, at 10:58 AM, David Noriega wrote:
> I was checking out zfsonlinux.org to see how things have been going
> lately and I had a question. Whats the difference, or whats better:
> Use a hardware raid5(or 6) or use zfs to create a raidz pool? In terms
> of Lustre, is one preferred over another?
ZFS much prefers to have direct access to the individual disks in a JBOD,
instead of via h/w RAID-5/6.  There are several reasons:

- it "knows" where the data and parity are located, and if there is an
  error reading data from disk it can retry with different data/parity
  combinations until the checksum matches, even trying single-bit error
  recovery in extreme cases
- it is easier to locate multiple copies of the metadata on different
  disks and if it has direct access to the individual disks
- it has more IO queues and can schedule IO better for individual disks,
  keeping the IO queue relatively shallow so that read latency isn''t
hurt
- pooled storage, in theory, allows all space/bandwidth to be used by any
  thread doing IO.  In practice this doesn''t perform as well as in
theory.
- no read-modify-write when writing "partial block" data (there
isn''t really
  such a thing as a "partial block write" for RAID-Z"

The main drawback is that RAID-Z needs a lot more effort when rebuilding
a failed disk compared to a normal RAID-5/6.  ZFS proponents will claim
that "it only needs to rebuild the used parts of the filesystem", but
most HPC filesystems are kept 70-80% full, so the RAID-Z overhead wipes
out any advantage gained by not rebuilding the 20% of unused space.

See zfsonlinux.org/docs/SC10_BoF_ZFS_on_Linux_for_Lustre.pdf for some
performance comparisons.


Cheers, Andreas
--
Andreas Dilger 
Principal Engineer
Whamcloud, Inc.