zfs discuss - Feb 2007 - Does running redundancy with ZFS use as much disk space as doubling drives?

If I''m gonna use OpenSolaris, I will have to buy new hardware, which I
can''t really defend at the moment. But I may be able to defend it in
the near future if using redundancy on ZDF uses less disk space as simply
getting extra drives and do identical copies, with periodic CRC checks of the
source material to check the health. Or will I use as much drive space with
redundancy? I''m more confused then I usually am...

Thanks in advance!
Tor
 
 
This message posted from opensolaris.org

On Mon, Feb 26, 2007 at 01:53:17AM -0800, Tor wrote:
> [...] if using redundancy on ZDF
The ZFS Document Format?  ;-)
> uses less disk space as simply getting extra drives and do identical
copies,
> with periodic CRC checks of the source material to check the health.
If you create a 2-disk mirror, then it is indeed simply two copies.
But if you create, say, a 5-disk RAID-Z group, then you get 4 data
disks worth of space.

Jeff

> My plan was to have 8-10 cheap drives, most of them IDE drives from
> 120 gig and up to 320 gig. Does that mean that I can get 7-9 drives
> with data plus full redundancy from the last drive? It sounds almost
> like magic to me to be able to have the data on maybe 1 TB of drives
> and have one drive that gives redundancy.
RAID does indeed seem almost magical, but it really does work.
The basic idea is simple.  Suppose you have 4 data drives, A, B, C, D.
You want to be able to survive the loss of any drive, so you add a
fifth drive, E, and generate the data for it using this equation:

    E = A ^ B ^ C ^ D

where ^ is XOR.  You can do it with addition too, but XOR is nicer
because there''s no carry bit to worry about.  It also has the nice
property that A ^ A = 0, so any number is its own negative with
respect to XOR.  In particular, XOR''ing E with both sides, the
equation above can be written in a pleasingly symmetic form:

    A ^ B ^ C ^ D ^ E = 0

The key is that you can reconstruct any drive from this equation.
For example, suppose you lose drive B.  No problem:

    B = A ^ C ^ D ^ E

and similarly for all the others.

Pretty cool, eh?  FYI, RAID was invented by David Patterson in 1988.
It''s hard to imagine life without it, yet it''s only 19 years
old --
still a relatively new idea in the grand scheme of things.

Jeff

Also note that you may not need new hardware. While not as compatible as linux,
it will run on a variety of hardware, so you may just be able to get by with new
disks.
 
 
This message posted from opensolaris.org

In other words, say you have 4 - 500 GB drives. In a standard raidz
configuration, you should yield (4-1) * 500 GB of space, or 1.5 TB.

In your case, I will mention one caveat. Say you have 8 drives in raidz. If you
have 7 500 GB and 1 20GB drive, you will only yield (8-1) * 20GB or 140GB of
space, which would be horrible space utilization. This is because zfs needs to
use the smallest drive to base the raid''s size off of. This is a
physical limitation of raid since the data is striped across all disks in the
group.
Example:
120GB + 120GB + 160GB + 160GB + 320GB + 320GB + 320GB + 500GB = (8-1) * 120GB =
840GB (1060GB unusable)

However, if you group the drives together into similar sized groups, you could
have much better results while reducing unusable space on your drives. Also, by
grouping into say two sets of 4 disks instead of one set of 8, you could grow
the raid in increments of 4 disks rather than 8.
For example:
120GB + 120GB + 160GB + 160GB = (4-1) * 120GB = 360GB (80GB unusable)
320GB + 320GB + 320GB + 500GB = (4-1) * 320GB = 960GB (180GB unusable)

This would ''waste'' much less space than putting them into one
big group. Also, these can all be in the same pool, so you would not have to
worry about splitting files or different mounts/partitions/etc.

These can be setup in the same pool and this pool can be grown at any time by
adding 4 more disks (upto as many drives as your hardware can support). These
new disks can be any size available. If all 4 new drives are the same size,
there will be zero wasted space too.

Eric
 
 
This message posted from opensolaris.org