zfs discuss - Jan 2008 - ZFS Home NAS - Configuration & Questions

Hi,

I am new to Solaris, but intrigued by ZFS. I am planning to set up a home NAS
(SAMBA/CIFS on ZFS) with my rough plan being to boot SXDE from an IDE drive,
then set up a single storage pool with 4 SATA drives (2 x 250GB & 2 x 500GB)
on a single controller.

My main concerns are redundancy (1-2 parity) & maximum storage size; I am
not concerned about performance.

I had a couple of questions regarding this:
1. Considering the drives are different sizes, would I be better off setting up
2 x 2-way mirrors separately and then adding them to the pool?
2. If I use RAIDZ instead of mirroring, will I always be protected against a
single disk failure (even though the disks are different sizes)?
3. With RAIDZ, is there anyway to determine where an individual file resides?
Specifically, is there any way to determine whether the parity information for
the file (or the User Copy) is residing on a separate device/drive? I know that
ZFS ''tries'' to do it, but can you check?
4. Assume that 12 months later I want to remove the smaller drives and replace
them with larger (TB) drives. Is it easy enough to remove them (presumably one
at a time) without losing any data?
5. Can anyone recommend a cheap (but reliable) SATA PCI or PCIX card? Again,
performance is not that important.

Thanks in advance.
 
 
This message posted from opensolaris.org

On Jan 24, 2008 8:07 AM, Kava <kava_kicks at yahoo.com.au>
wrote:
> 1. Considering the drives are different sizes, would I be better off
setting up 2 x 2-way mirrors separately and then adding them to the pool?
Yes.  With raidz you will only get the capacity of the smallest disk
times (number of disks - 1), or 750 GB.  Since this is the same as a
pair of mirrors, you might as well use the mirrors, which will likely
perform better.
> 2. If I use RAIDZ instead of mirroring, will I always be protected against
a single disk failure (even though the disks are different sizes)?
Yes.
> 3. With RAIDZ, is there anyway to determine where an individual file
resides? Specifically, is there any way to determine whether the parity
information for the file (or the User Copy) is residing on a separate
device/drive? I know that ZFS ''tries'' to do it, but can you
check?
With raidz every file is striped across all your drives.  Writes
happen in $blocksize chunks, up to a maximum of 128k.  With a raidz
vdev these 128k chunks are broken into (number of devices - 1) pieces,
and parity is generated from those N-1 pieces.  Then the new set of N
pieces are written to your N drives.
> 4. Assume that 12 months later I want to remove the smaller drives and
replace them with larger (TB) drives. Is it easy enough to remove them
(presumably one at a time) without losing any data?
Yes.  "cfgadm -c unconfigure old-device", then hot-unplug it and plug
the new drive in and run "zpool replace old-device".
> 5. Can anyone recommend a cheap (but reliable) SATA PCI or PCIX card?
Again, performance is not that important.
I''m using the Supermicro-branded AOC-SAT2-MV8 with the Marvell
88SX6081 chipset.  It''s worked well for me, except the one defective
card I got (look at the archives for details on what happened - no
data loss, just annoyingly slow).

Will

OK, I have been reading a bit more and I think I can answer some of these
questions.
For those interested, read page 55 of the Solaris ZFS Administration Guide.

Q4: "Replace a disk or disks in an existing mirrored configuration as long
as the replacement disks are greater than or equal to the device to be
replaced."

Q1: Considering you can add more mirrored virtual devices to the pool as you go,
I am thinking that adding the drives as 2 by 2-way mirrors makes more sense. I
think this would make it easier to move them as well, cause you just add a new 2
by 2-way mirror with 2 bigger drives and migrate the old mirror to it.

That said, I am still not 100% sure about the differences between Option 1 &
2 below. Both have two parity bits, so they effectively provide the same level
of protection (assuming RAIDZ does a good job of spreading the data around).
Disk space is the same. Not sure about performance or manageability though ...
Option 1: One four-way RAID-Z2 configuration
Option 2: Two two-way mirrors using dynamic striping
 
 
This message posted from opensolaris.org

More info from the same guide, page 59: "The command also warns you about
creating a mirrored or RAID-Z pool using devices of different sizes. While this
configuration is allowed, mismatched levels of redundancy result in unused space
on the larger device, and requires the -f option to override the warning."

Given the above, I am guessing that my options are actually different than I
initially thought.

Option 1: One four-way RAID-Z2 configuration. This is no longer possible because
the drives are of a different size and you can not create a RAIDZ2 or even a
RAIDZ configuration with only 2 drives.

Option 2: Two two-way mirrors using dynamic striping. Still possible.

Another option would be:
Option 3: One 4 disk stripe, with 2 User Copies running. I guess the downside of
this is you don''t really know where the copies end up (at least I
don''t think you do).
 
 
This message posted from opensolaris.org

Kava <kava_kicks at yahoo.com.au> wrote:
> Can anyone recommend a cheap (but reliable) SATA PCI or PCIX card?
Why would you get a PCI-X card for a home NAS? I don''t think
I''ve ever
seen a non-server motherboard with PCI-X. Are you sure you don''t want a
PCI-E card instead?

Anyway, if someone is aware of some cheap (but reliable) SATA PCI-E
card then I''d be very interested.


- Marcus

Marcus Sundman wrote:
> Kava <kava_kicks at yahoo.com.au> wrote:
>   
>> Can anyone recommend a cheap (but reliable) SATA PCI or PCIX card?
>>     
>
> Why would you get a PCI-X card for a home NAS? I don''t think
I''ve ever
> seen a non-server motherboard with PCI-X. Are you sure you don''t
want a
> PCI-E card instead?
>
> Anyway, if someone is aware of some cheap (but reliable) SATA PCI-E
> card then I''d be very interested.
>   
It may be less expensive to purchase a new motherboard with 6 SATA
ports on it.
 -- richard

Ok I am not an expert - just done some playing about.

Option 1) I have done this - i had 4x300gb disks and one more in the post, i
could not wait to build my raidz2 so i used a 73gb which was spare - what it
gave me was a raidz2 pool of 5x73gb.
The warning is there to ask you if you really want to lose all the space.

zpool replace 73gb 300gb disk worked fine and i went to a pool with 5x300gb

Option 2) Done this too with some 18gb mirror and a 73gb mirror, seem to work
fine no performance drops.  I guess the stripe will only last for 18gb then drop
to a single disk performance

Option 3) Not sure about this one, not had chance to play with this feature

Hope this helps a bit

Also see http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide
 
 
This message posted from opensolaris.org

I use the supermicro 8 port pci-x card, its about 70 pounds in the uk

Works fine on my home nas box which uses the Asus M2N32 WS Pro, which i can use
6 sata of the nvidia chipset giving me 14 usable sata with the solaris native
sata support
 
 
This message posted from opensolaris.org

Thanks.

I am going to try this (replacement with larger drive) again ... it sounds damn
handy and I am pretty sure I must have done something wrong ...
 
 
This message posted from opensolaris.org

That is a lot of drives ;)
 
 
This message posted from opensolaris.org

I finally got this to work, but it did not happen automatically. I needed to
export then re-import the pool to get it to work. Only then did the additional
space appear.

Here is what I did:
- create 4 x 8GB disks and 1 x 4 GB disks
- create RAIDZ pool with 3 x 8GB disks & 1 x 4GB
- ignore warning about wasted space
- end up with pool of 12GB usable (4 x 4GB less 4GB parity)
- replace 4GB disk with 8GB disk
- check pool stats and note that it still has 12GB usable space
- export pool (use defaults)
- import pool (use defaults)
- check pool stats and note that usable space is now 24GB 

Is there a way to skip the export/import?? Downtime for a home NAS is not really
that big an issue ... but if you can do it, I would like to know how.
 
 
This message posted from opensolaris.org

Yep - don''t ever get into digital media :(
 
 
This message posted from opensolaris.org

Kava wrote:
> I finally got this to work, but it did not happen automatically. 
> I needed to export then re-import the pool to get it to work. 
>  Only then did the additional space appear.
> 
> Here is what I did:
> - create 4 x 8GB disks and 1 x 4 GB disks
> - create RAIDZ pool with 3 x 8GB disks & 1 x 4GB
> - ignore warning about wasted space
> - end up with pool of 12GB usable (4 x 4GB less 4GB parity)
> - replace 4GB disk with 8GB disk
> - check pool stats and note that it still has 12GB usable space
> - export pool (use defaults)
> - import pool (use defaults)
> - check pool stats and note that usable space is now 24GB 
> 
> Is there a way to skip the export/import?? Downtime for a home
> NAS is not really that big an issue ... but if you can do it,
> I would like to know how.


hi Kava,
I''m using mirrors in my zpools on snv_77, and recently upgraded
the disks. I first replaced one 200Gb disk with a 320Gb using

# zpool replace -f soundandvision c2t1d0s0


and after less than an hour the data had finished re-silvering.
Then the next day when I was able to reboot my system, I replaced
the other <320Gb disk, did another in-place replacement:

# zpool replace -f soundandvision c3t1d0s0


At that point, I was all set to export the pool, but realised
that I didn''t need to because zfs had picked up that the disks
were larger - an extra ~100Gb just showed up.

I don''t know whether using a raidZ or raidZ2 pool requires the
export/import routine, but I hope it doesn''t.


cheers,
James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
http://blogs.sun.com/jmcp	http://www.jmcp.homeunix.com/blog

Richard Elling <Richard.Elling at Sun.COM> wrote:
> Marcus Sundman wrote:
> > Kava <kava_kicks at yahoo.com.au> wrote:
> >   
> >> Can anyone recommend a cheap (but reliable) SATA PCI or PCIX card?
> >>     
> >
> > Why would you get a PCI-X card for a home NAS? I don''t think
I''ve
> > ever seen a non-server motherboard with PCI-X. Are you sure you
> > don''t want a PCI-E card instead?
> >
> > Anyway, if someone is aware of some cheap (but reliable) SATA PCI-E
> > card then I''d be very interested.
> 
> It may be less expensive to purchase a new motherboard with 6 SATA
> ports on it.
Sure, but which one? I''ve been trying to find one for many, many months
already, but it has turned out to be impossible to find anyone that has
tried one successfully and is willing to tell about it. I''m currently
looking at the NForce 570 SLI based GA-M57SLI-S4 motherboard, but I
don''t know if it''ll work well or not. If someone knows of some
common
(and cheap) motherboard with 6+ SATA ports and gigabit ethernet then
please tell.

Also, if I just use on-board ports then:
A) I''m stuck with as many ports as the motherboard has, and
B) when the manufacturer stops making the motherboard I have to start
my search all over again (and mobos become obsolete much, much quicker
than expansion cards).
However, although I''d rather find an inexpensive SATA PCI-E card, a
mobo
with 6+ SATA ports is just fine for me at this point.


- Marcus

Marcus Sundman wrote:
> Richard Elling <Richard.Elling at Sun.COM> wrote:
>> Marcus Sundman wrote:
>>> Kava <kava_kicks at yahoo.com.au> wrote:
>>>   
>>>> Can anyone recommend a cheap (but reliable) SATA PCI or PCIX
card?
>>>>     
>>> Why would you get a PCI-X card for a home NAS? I don''t
think I''ve
>>> ever seen a non-server motherboard with PCI-X. Are you sure you
>>> don''t want a PCI-E card instead?
>>>
>>> Anyway, if someone is aware of some cheap (but reliable) SATA PCI-E
>>> card then I''d be very interested.
>> It may be less expensive to purchase a new motherboard with 6 SATA
>> ports on it.
> 
> Sure, but which one? I''ve been trying to find one for many, many
months
> already, but it has turned out to be impossible to find anyone that has
> tried one successfully and is willing to tell about it.
ASUS M2N-E works fine.



-- 
Kaiser Jasse -- Authorized Stealth Oracle

The axioms of wisdom:
1. Go the SPARC way of life
2. You can''t conquer the universe without the knowledge of FORTRAN
3. In the Unix realm, 10% of work fixes 90% of the problems

Marcus Sundman <sundman at iki.fi> wrote:
> Richard Elling <Richard.Elling at Sun.COM> wrote:
> > It may be less expensive to purchase a new motherboard with 6 SATA
> > ports on it.
> 
> Sure, but which one? I''ve been trying to find one for many, many
> months already, but it has turned out to be impossible to find anyone
> that has tried one successfully and is willing to tell about it.
I''m
> currently looking at the NForce 570 SLI based GA-M57SLI-S4
> motherboard, but I don''t know if it''ll work well or not.
Hah! Funny timing. After I wrote that, but before I pressed the
Send-button I got a message from Nathan Kroenert (thanks, Nathan!)
telling me that the GA-M57SLI-S4 works very well indeed. I think I''ll
order one of those tonight.


- Marcus

Marcus:

I''m currently running the asus K8N-LR, and it works wonderfully.  Not
only do the onboard ports work, but it also has multiple pci-x slots. 
I''m running an opteron 165 (dual core) cpu with it.  It''s
cheap, and fast.
http://usa.asus.com/products.aspx?l1=9&l2=39&l3=263&l4=0&model=1023&modelmenu=1

You can then pick up the supermicro sata cards here:
http://www.ewiz.com/detail.php?p=AOC-SAT2MV&c=fr&pid=84b59337aa4414aa488fdf95dfd0de1a1e2a21528d6d2fbf89732c9ed77b72a4

I''m currently running a 10-disk pool with plenty of room to grow :)
 
 
This message posted from opensolaris.org

Oh, one thing.  The only downside is the onboard gigE interfaces are the
broadcom pci-e based nic''s.  They unfortunately do not support jumbo
frames.  I doubt this will be an issue for you if it''s just a home NAS.
In my setup I''ve pushed 50MB/sec over nfs and the server was barely
breathing.
 
 
This message posted from opensolaris.org

Tim Cook <tim at tcsac.net> wrote:
> I''m currently running the asus K8N-LR, and it works wonderfully.
Thanks, but socket 939 is cold dead and buried. S939 CPUs are very
expensive. DDR is over twice as expensive as DDR2. I can''t tell if the
motherboard is expensive or not because I just can''t find it _anywhere_
in Finland (and is thus definitely not "common", which was one of my
few
criterias). And to top it all off it has one of those whiny chipset
fans, which is bad both for a _home_ nas (because of the noise) and for
reliability (because when one breaks the heatsink it''s attached to is
way too small for the general air flow in the case).


Regards,
Marcus

I have an older server that I am using (2 x 2Gb Xeon) .. used ot run a web
company ;0
 
 
This message posted from opensolaris.org

Hmm .. the only things that I am doing differently (as far as I can tell) is I
am using the web GUI to issue the commands and I am running it in a VM (using
the pre-buil ones available).

I will try using the command line and see if that makes a difference. Maybe I
wasn''t waiting for it to resilver ... though there is no data on the
drives so it looked like it was complete.
 
 
This message posted from opensolaris.org