zfs discuss - Jul 2008 - Case study/recommended ZFS setup for home file server

Hello,

I plan to use (Open)Solaris for a home file server.  I wanted cool and
quiet hardware, so I picked a mini-atx motherboard and case, an AMD64
CPU and 4 GB of RAM.  My case has room for three hard drives and I
have chosen 3x WD 750 Green Power hard drives.  The file server will
serve out via NFS and Samba the home directories, the library
(collected articles and books in PDF format) and the photo archive
(150GB and growing of photos in RAW format ~ 7-9MB/file).

I cannot use OpenSolaris 2008.05 since it does not recognize the SATA
disks attached to the southbridge. A fix for this problem went into
build 93.  I will use SXCE 93 (for the SATA fix) or SXCE 94 (for the
last revision of the ZFS format).

In order to make the maximum amount of space available for the photos,
I plan to use RAID-5 for that pool.  Also, I would like to have
sufficient redundancy so if a drive goes bad, I can just replace it
and the volume manger/file system will take care of fixing itself
back.

The question is, how should I partition the drives, and what tuning
parameters should I use for the pools and file systems?  From reading
the best practices guides [1], [2], it seems that I cannot have the
root file system on a RAID-5 pool, but it has to be a separate storage
pool.  This seems to be slightly at odds with the suggestion of using
whole-disks for ZFS, not just slices/partitions.

My plan right now is to create a 20 GB and a 720 GB slice on each
disk, then create two storage pools, one RAID-1 (20 GB) and one RAID-5
(1.440 TB).  Create the root, var, usr and opt file systems in the
first pool, and home, library and photos in the second.  I hope I
won''t need swap, but I could create three 1 GB slices (one on each
disk) for that.

Does this sound like a good configuration?

Will the SXCE 9[34] installer allow me to create the above setup?

Should I pass any special parameters to the zfs pool and file system
creation tool to get the best performance?  home and library contains
files between few KB and a fer MB.  photos contains file roughly 7 to
9 MB.  Should I place those on separate pools?

Note: the hardware is committed (i.e. I already have it), so I am not
inclined to deviate from it 8^)

Thanks,
florin

1: http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide
2: http://www.solarisinternals.com/wiki/index.php/ZFS_Configuration_Guide

-- 
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080709/a79b3f0c/attachment.bin>

On Wed, Jul 9, 2008 at 3:37 PM, Florin Iucha <florin at iucha.net>
wrote:
> The question is, how should I partition the drives, and what tuning
> parameters should I use for the pools and file systems?  From reading
> the best practices guides [1], [2], it seems that I cannot have the
> root file system on a RAID-5 pool, but it has to be a separate storage
> pool.  This seems to be slightly at odds with the suggestion of using
> whole-disks for ZFS, not just slices/partitions.
The reason for using a whole disk is that ZFS will turn on the drive''s
cache. When using slices, the cache is normally disabled. If all
slices are using ZFS, you can turn the drive cache back on. I don''t
think it happens by default right now, but you can set it manually.

Another alternative is to use an IDE to Compact Flash adapter, and
boot off of flash. I''ll be building a media server once we move, and
that system will boot from flash. You can also boot from USB keys, but
USB under OpenSolaris seems to be iffy.

Here''s the component list that I''m planning to use right now:
http://secure.newegg.com/WishList/PublicWishDetail.aspx?Source=MSWD&WishListNumber=7739092

I *may* change it and boot off another drive that is not part of the
RAID-Z pool.
> My plan right now is to create a 20 GB and a 720 GB slice on each
> disk, then create two storage pools, one RAID-1 (20 GB) and one RAID-5
> (1.440 TB).  Create the root, var, usr and opt file systems in the
> first pool, and home, library and photos in the second.  I hope I
> won''t need swap, but I could create three 1 GB slices (one on each
> disk) for that.
>
> Does this sound like a good configuration?
If you have enough memory (say 4gb) you probably won''t need swap. I
believe swap can live in a ZFS pool now too, so you won''t necesarily
need another slice. You''ll just have RAID-Z protected swap.

I built a Linux-based NAS a few years back using an almost identical
scheme and wound up regretting it. In the future I would install the
system on a completely separate disk or group of disks than the shared
pool.
> Should I pass any special parameters to the zfs pool and file system
> creation tool to get the best performance?  home and library contains
> files between few KB and a fer MB.  photos contains file roughly 7 to
> 9 MB.  Should I place those on separate pools?
You shouldn''t need to do anything. If you want to set the block size,
or enable or disable compression, etc. you can create multiple
filesystems in your pool rather than multiple pools.
> Note: the hardware is committed (i.e. I already have it), so I am not
> inclined to deviate from it 8^)
You might want to look at a 4 or 8 port SATA adapter rather than wait
for the southbridge fixes.

-B

-- 
Brandon High bhigh at freaks.com
"The good is the enemy of the best." - Nietzsche

Thank you for starting this thread. I hope you get some good feedback.

Your questions are quite frequently asked in this forum, but I''m very
interested in the topic. Anway, the "best" answer varies from month
to month. So I hope you get some good feedback.
> I cannot use OpenSolaris 2008.05 since it does not
> recognize the SATA disks attached to the southbridge.
> A fix for this problem went into build 93.
Which forum/mailing list discusses SATA issues like the above?

Thanks
 
 
This message posted from opensolaris.org

On Wed, Jul 09, 2008 at 08:42:37PM -0700, Bohdan Tashchuk
wrote:
> > I cannot use OpenSolaris 2008.05 since it does not
> > recognize the SATA disks attached to the southbridge.
> > A fix for this problem went into build 93.
> 
> Which forum/mailing list discusses SATA issues like the above?
#opensolaris in freenode.net

I booted from the OpenSolaris LiveCD/installer, and noticing the lack
of available disks, I cried for help on #irc.  There were a few
helpful people that gave me some commands to run, to try and get this
going.  After their efforts failed, I googled for solaris and SB600
(this is the ATI SouthBridge chip) and found a forum posting from
another user, back in February, and the hit in bugzilla, pointing to
the resolution of the bug, with the target being snv_93.

Cheers,
florin

-- 
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080709/9cdce43b/attachment.bin>

On Wed, Jul 09, 2008 at 06:02:24PM -0700, Brandon High
wrote:
> On Wed, Jul 9, 2008 at 3:37 PM, Florin Iucha <florin at iucha.net>
wrote:
> The reason for using a whole disk is that ZFS will turn on the
drive''s
> cache. When using slices, the cache is normally disabled. If all
> slices are using ZFS, you can turn the drive cache back on. I
don''t
> think it happens by default right now, but you can set it manually.
Aha! Good to know.
> Another alternative is to use an IDE to Compact Flash adapter, and
> boot off of flash. I''ll be building a media server once we move,
and
> that system will boot from flash. You can also boot from USB keys, but
> USB under OpenSolaris seems to be iffy.
> 
> Here''s the component list that I''m planning to use right
now:
>
http://secure.newegg.com/WishList/PublicWishDetail.aspx?Source=MSWD&WishListNumber=7739092
That adapter won''t work for me, since I have a single IDE port, and I
need to use the DVD to install the OS and maybe to run some backups.

However, this looks interesting:

   http://www.addonics.com/products/flash_memory_reader/ad2sahdcf.asp

as it has hardware mirroring.  Not sure what the error reporting
through the OS is though..., but I hope I don''t have to find out.

For the Compact Flash I would spring for the industrial grade:

  
http://www.hitechvendors.com/showproduct.aspx?ProductID=4885&SEName=transcend-4gb-100x-industrial-cf-card-udma4-mode
> > My plan right now is to create a 20 GB and a 720 GB slice on each
> > disk, then create two storage pools, one RAID-1 (20 GB) and one RAID-5
> > (1.440 TB).  Create the root, var, usr and opt file systems in the
> > first pool, and home, library and photos in the second.  I hope I
> > won''t need swap, but I could create three 1 GB slices (one on
each
> > disk) for that.
> 
> I built a Linux-based NAS a few years back using an almost identical
> scheme and wound up regretting it. In the future I would install the
> system on a completely separate disk or group of disks than the shared
> pool.
This is the current Linux-based NAS and I''m not happy with its
performance, either.
> > Note: the hardware is committed (i.e. I already have it), so I am not
> > inclined to deviate from it 8^)
> 
> You might want to look at a 4 or 8 port SATA adapter rather than wait
> for the southbridge fixes.
I like the southbridge since it sits on the PCI express bus.  The PCI bus
is limited to 133 MB/s, which divided by 3 (disks) means 35-40 MB/s
(including overhead) writes.  And good quality PCI-express add-on
controllers with Solaris drivers are quite expensive.

Cheers,
florin

-- 
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080709/2a060fcd/attachment.bin>

Florin Iucha wrote:
> On Wed, Jul 09, 2008 at 08:42:37PM -0700, Bohdan Tashchuk wrote:
>   
>>> I cannot use OpenSolaris 2008.05 since it does not
>>> recognize the SATA disks attached to the southbridge.
>>> A fix for this problem went into build 93.
>>>       
>> Which forum/mailing list discusses SATA issues like the above?
>>     
>
> #opensolaris in freenode.net
>
> I booted from the OpenSolaris LiveCD/installer, and noticing the lack
> of available disks, I cried for help on #irc.  There were a few
> helpful people that gave me some commands to run, to try and get this
> going.  After their efforts failed, I googled for solaris and SB600
> (this is the ATI SouthBridge chip) and found a forum posting from
> another user, back in February, and the hit in bugzilla, pointing to
> the resolution of the bug, with the target being snv_93.
>   
The OpenSolaris live CD has a hardware device detection tool.
Please run it and submit the results (everyone should do this :-)

b93 should be out soon, in the next week or so.
 -- richard

Florin Iucha wrote:
> On Wed, Jul 09, 2008 at 08:42:37PM -0700, Bohdan Tashchuk wrote:
>>> I cannot use OpenSolaris 2008.05 since it does not
>>> recognize the SATA disks attached to the southbridge.
>>> A fix for this problem went into build 93.
>> Which forum/mailing list discusses SATA issues like the above?
> 
> #opensolaris in freenode.net
> 
> I booted from the OpenSolaris LiveCD/installer, and noticing the lack
> of available disks, I cried for help on #irc.  There were a few
> helpful people that gave me some commands to run, to try and get this
> going.  After their efforts failed, I googled for solaris and SB600
> (this is the ATI SouthBridge chip) and found a forum posting from
> another user, back in February, and the hit in bugzilla, pointing to
> the resolution of the bug, with the target being snv_93.

For reference of other people, the bug in question is

http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6665032
6665032 ahci driver doesn''t work for ATI SB600 AHCI chipset (ASUS
M2A-VM)

which is fixed in snv_93




James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
http://blogs.sun.com/jmcp	http://www.jmcp.homeunix.com/blog

My recommendation:  buy a small, cheap 2.5" SATA hard drive (or 1.8"
SSD) and use that as your boot volume, I''d even bolt it to the side of
your case if you have to.  Then use the whole of your three large disks as a
raid-z set.

If I were in your shoes I would also have bought 4 drives for ZFS instead of 3,
and gone for raid-z2.

And finally, I don''t know how much room you have in your current case,
but if you''re ever looking for one that takes more drives I can highly
recommend the Antec P182.  I''ve got 6x 1TB drives in my home server and
in that case it''s so quiet I can''t even hear it turn on.  My
watch ticking easily drowns out this server.

PS.  If you''re going to be using CIFS, avoid build 93.
 
 
This message posted from opensolaris.org

Brandon High wrote:
> On Wed, Jul 9, 2008 at 3:37 PM, Florin Iucha <florin at iucha.net>
wrote:
>   
>> The question is, how should I partition the drives, and what tuning
>> parameters should I use for the pools and file systems?  From reading
>> the best practices guides [1], [2], it seems that I cannot have the
>> root file system on a RAID-5 pool, but it has to be a separate storage
>> pool.  This seems to be slightly at odds with the suggestion of using
>> whole-disks for ZFS, not just slices/partitions.
>>     
>
> The reason for using a whole disk is that ZFS will turn on the
drive''s
> cache. When using slices, the cache is normally disabled. If all
> slices are using ZFS, you can turn the drive cache back on. I
don''t
> think it happens by default right now, but you can set it manually.
>
>   
As I recall, using whole disk as zfs also change the disk label to EFI. 
Meaning, you can''t boot from it.
> Another alternative is to use an IDE to Compact Flash adapter, and
> boot off of flash.
Just curious, what will that flash contain?
e.g. will it be similar to linux''s /boot, or will it contain the full 
solaris root?
How do you manage redundancy (e.g. mirror) for that boot device?
>> My plan right now is to create a 20 GB and a 720 GB slice on each
>> disk, then create two storage pools, one RAID-1 (20 GB) and one RAID-5
>> (1.440 TB).  Create the root, var, usr and opt file systems in the
>> first pool, and home, library and photos in the second.  
Good plan.
>> I hope I
>> won''t need swap, but I could create three 1 GB slices (one on
each
>> disk) for that.
>>
>>     
> If you have enough memory (say 4gb) you probably won''t need swap.
I
> believe swap can live in a ZFS pool now too, so you won''t
necesarily
> need another slice. You''ll just have RAID-Z protected swap.
>   
Really? I think solaris still needs non-zfs swap for default dump device.

Regards,

Fajar
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3242 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080710/5214e65c/attachment.bin>

Fajar A. Nugraha wrote:
>> If you have enough memory (say 4gb) you probably won''t need
swap. I
>> believe swap can live in a ZFS pool now too, so you won''t
necesarily
>> need another slice. You''ll just have RAID-Z protected swap.
>>   
> Really? I think solaris still needs non-zfs swap for default dump device.
No longer true, you can swap and dump to a ZVOL (but not the same one). 
  This change came in after OpenSolaris 2008.05 LiveCD/Install was cut 
so it doesn''t take advantage of that.   There was a big long thread 
cross posted to this list about it just recently.

The current SX:CE installer (ie Nevada) uses ZVOL for swap and dump.

-- 
Darren J Moffat

On Thu, Jul 10, 2008 at 12:47:26AM -0700, Ross wrote:
> My recommendation:  buy a small, cheap 2.5" SATA hard drive (or
1.8" SSD) and use that as your boot volume, I''d even bolt it to
the side of your case if you have to.  Then use the whole of your three large
disks as a raid-z set.
Yup, I''m going with 4GB of mirrored flash for root/var/usr and
I''ll keep
the main spindles only for data.
> If I were in your shoes I would also have bought 4 drives for ZFS instead
of 3, and gone for raid-z2.
No room - Antec NSK-2440 - and too much power draw.  My server idles at
57-64 W (under Linux) and I''d like to keep it that way.
> And finally, I don''t know how much room you have in your current
case, but if you''re ever looking for one that takes more drives I can
highly recommend the Antec P182.  I''ve got 6x 1TB drives in my home
server and in that case it''s so quiet I can''t even hear it
turn on.  My watch ticking easily drowns out this server.
Heh - I do have the P180 as my workstation case.  But I don''t have
that much room for servers 8^)
> PS.  If you''re going to be using CIFS, avoid build 93.
Can you please give a link to the discussion, or a bug id?

Thanks,
florin

-- 
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080710/9b629ead/attachment.bin>

Fajar A. Nugraha wrote:
> Brandon High wrote:
>> Another alternative is to use an IDE to Compact Flash adapter, and
>> boot off of flash.
> Just curious, what will that flash contain?
> e.g. will it be similar to linux''s /boot, or will it contain the
full
> solaris root?
> How do you manage redundancy (e.g. mirror) for that boot device?
>
zfs set copies=2 :-)


hmm... I need to dig up my notes on that and blog it...
 -- richard

It was posted in the CIFS forum a couple of days ago:
http://www.opensolaris.org/jive/forum.jspa?forumID=214

Thread: HEADS-UP: Please skip snv_93 if you use CIFS server:
http://www.opensolaris.org/jive/thread.jspa?threadID=65996&tstart=0
 
 
This message posted from opensolaris.org

On Thu, Jul 10, 2008 at 1:15 AM, Fajar A. Nugraha <fajar at fajar.net>
wrote:
>> Another alternative is to use an IDE to Compact Flash adapter, and
>> boot off of flash.
>
> Just curious, what will that flash contain?
> e.g. will it be similar to linux''s /boot, or will it contain the
full
> solaris root?
> How do you manage redundancy (e.g. mirror) for that boot device?
4gb is enough to hold a minimal system install. /var will go to a file
system on the raidz pool.

ZFS mirroring can be used on boot devices for redundancy.

-B

-- 
Brandon High bhigh at freaks.com
"The good is the enemy of the best." - Nietzsche

On Wed, Jul 09, 2008 at 11:18:23PM -0500, Florin Iucha
wrote:
> On Wed, Jul 09, 2008 at 06:02:24PM -0700, Brandon High wrote:
> > Here''s the component list that I''m planning to use
right now:
> >
http://secure.newegg.com/WishList/PublicWishDetail.aspx?Source=MSWD&WishListNumber=7739092
> 
>  this looks interesting:
> 
>    http://www.addonics.com/products/flash_memory_reader/ad2sahdcf.asp
> 
> as it has hardware mirroring.  Not sure what the error reporting
> through the OS is though..., but I hope I don''t have to find out.
> 
> For the Compact Flash I would spring for the industrial grade:
> 
>   
http://www.hitechvendors.com/showproduct.aspx?ProductID=4885&SEName=transcend-4gb-100x-industrial-cf-card-udma4-mode
I got the adapter and the flash, and SXCE 93 does not like them.

When I have the adapter set to RAID1, the installer sees the device
and it hangs afterwards, but the spinner is active.  I let it for 30
minutes and it''s still spinning.  If I take one of the cards out and
set it up in concatenation, the installer gives an "AHCI taskfile
error" and proceeds, but later on ''format'' does not see
the disk.

Centos saw the AHCI southbridge controller and CF-SATA adapter without
problem and it installed quite happily.

I guess no ZFS for me... 8-((

florin

-- 
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080717/d2eb9775/attachment.bin>