freebsd stable - Apr 2008 - [SO]HO Software RAID5 server: which implementation should I choice?

Hello, freebsd-stable.

  Does somebody use some software RAID5 on FreeBSD in real production
  system?

  I want to build storage server for my home: RAW photos, multi-layer
PhotoShop files and FLAC-encoded music consume a lot of space, and
they should be availible both from desktop & notebook.

  Also, all photo-content is unique, so I need some insuranse from
single HDD crash. I understand, that I will not safe from fire, PSU
failure and thing slike this.

  I selected hardware platform: Intel Q35-based MoBo with 6xSATA-II ports
(all of them is chipset-based, so no SiliconImage/JMicron/Whatever
crappy controllers), some low-end Core2Duo, 2Gb of memory.
  Storage will be 5x500Gb WD HDDs for RAID + one small HDD for boot, system,
swap, etc. I want to have 2Tb (ok, not real Tb, I know) of "protected"
storage.
  I want to have maximum speed via 1Gb network, because graphic files
are big and should open fast. Not as fast as local ones, I understand
that, but speeds about 12-15Mb/s is not enough for sure :)

  Only problem I see: which software RAID5 solution should I prefer?
FreeBSD-based, of course!

  I see these variants:

 (1) FreeBSD 6(7?) + graid3. Slow, one disk for checksums is bottleneck,
     as far as I understand.

 (2) FreeBSD 6(7?) + gvinum/radi5. Is it stable enough?! Is it complete?
     when I try it about 6 months ago in VMWare installation with 5
     virtual disks, I got panics and strange behaviour after
"crashing"
     one of virtual disks.

 (3) FreeBSD 6(7?) + graid5. Again, is it stable enough? There are
     THREE versions of it. Which one should I prefer? There was long
     thread about it some times ago without any clear conclusion. Does
     something changed?

 (4) FreeBSD 7 + ZFS "zraid". And again: stability. Too many messages
     about locks, crashes, etc. Code is experemental. Is it only for
     32 bit systems?

 (5) Do I miss something?

 (6) Solaris + ZFS? I don't want it, I know a little about Solaris
     administaration, and I already have FreeBSD servers and routers.

  I know, that 3ware or Areca controllers are very good. I know, that
  "gmirror" is very stable. But these variants are too expensive for
  home server :(

  Does somebody use some software RAID5 on FreeBSD in real production?
  Any advices?

-- 
// Black Lion AKA Lev Serebryakov <lev@FreeBSD.org>

Lev Serebryakov a ?crit :
> Hello, freebsd-stable.
>
>   Does somebody use some software RAID5 on FreeBSD in real production
>   system?
>
>   I want to build storage server for my home: RAW photos, multi-layer
> PhotoShop files and FLAC-encoded music consume a lot of space, and
> they should be availible both from desktop & notebook.
>
>   Also, all photo-content is unique, so I need some insuranse from
> single HDD crash. I understand, that I will not safe from fire, PSU
> failure and thing slike this.
>
>   I selected hardware platform: Intel Q35-based MoBo with 6xSATA-II ports
> (all of them is chipset-based, so no SiliconImage/JMicron/Whatever
> crappy controllers), some low-end Core2Duo, 2Gb of memory.
>   Storage will be 5x500Gb WD HDDs for RAID + one small HDD for boot,
system,
> swap, etc. I want to have 2Tb (ok, not real Tb, I know) of
"protected"
> storage.
>   I want to have maximum speed via 1Gb network, because graphic files
> are big and should open fast. Not as fast as local ones, I understand
> that, but speeds about 12-15Mb/s is not enough for sure :)
>
>   Only problem I see: which software RAID5 solution should I prefer?
> FreeBSD-based, of course!
>
>   I see these variants:
>
>  (1) FreeBSD 6(7?) + graid3. Slow, one disk for checksums is bottleneck,
>      as far as I understand.
>
>  (2) FreeBSD 6(7?) + gvinum/radi5. Is it stable enough?! Is it complete?
>      when I try it about 6 months ago in VMWare installation with 5
>      virtual disks, I got panics and strange behaviour after
"crashing"
>      one of virtual disks.
>
>  (3) FreeBSD 6(7?) + graid5. Again, is it stable enough? There are
>      THREE versions of it. Which one should I prefer? There was long
>      thread about it some times ago without any clear conclusion. Does
>      something changed?
>
>  (4) FreeBSD 7 + ZFS "zraid". And again: stability. Too many
messages
>      about locks, crashes, etc. Code is experemental. Is it only for
>      32 bit systems?
>
>  (5) Do I miss something?
>
>  (6) Solaris + ZFS? I don't want it, I know a little about Solaris
>      administaration, and I already have FreeBSD servers and routers.
>
>   I know, that 3ware or Areca controllers are very good. I know, that
>   "gmirror" is very stable. But these variants are too expensive
for
>   home server :(
>
>   Does somebody use some software RAID5 on FreeBSD in real production?
>   Any advices?
>   
Hi !

I personally use the 3 option for my personal Home File Server. I got 
approximatly the same usage for the file server (mostly video, music, 
photo). I built my own about 12 month ago.
I reviewed the about the same variants as the one you propose :
(1) Discarded for performance issues. Raid3 is slow. Really.
(2) raid5/vinum is also slow. And as I understood at that time,  
recovery from lost hard drive wasn't easy enough for the freebsd niubee 
I was  then.
(4) ZFS wasn't there yet. But I did test it on a test VMWare, and wasn't
convinced (mostly stability and memory issues).

So I use geom_raid5. I sticked to the main distributions, which seemed 
more stable at the moment. The kernel module is fairly simple to 
build/install. Performance is (very) good for a software raid.
I successfully switched the raid array from an i386 6.2-RELEASE to an an 
amd64 7.0-RELEASE (with motherboard and CPU change) without any assle.

For the moment, I use one big UFS+SU (and snapshots) on the whole array. 
I successfuly tried unpplugging then replugging on of the drives, 
suddent power loss, using the array with a missing disk (degraded mode). 
All did work fine. (still, I use an UPS on the file server).

The sole issue I had is with ataidle. I had to patch ata-disc.c to 
increase the IO timeout. Without, the raid5 module detected temporary 
disk loss and constantly launched rebuilds of the array.

With 7.0, I wondered if I should use gjournal, but I'm not sure if it's 
really the way to go on a file system dedicated to store many big files. 
So I stick to soft updates.

Current configuration is :
/ on a 2GB usb key
/tmp on memory
ports and source trees (and some portsnap stuff) on a small disk
4x250 GB sata for the raid5 array.
AMD A64 3200+ and 512 GB DDRII
Realtek Gigabit nics.

Copy from raid5 to /dev/null gives about 100MB/s
Copy from /dev/random to raid5 about 40MB/s

I use samba shares. I get about 40MB/s in both ways from another 
computer on the network (enabling jumbo-frames gives a big boost).

Hope my own story can help you in any way.

Regards,

Arnaud Houdelette

On Wed, Apr 9, 2008 at 2:45 AM, Lev Serebryakov <lev@freebsd.org> wrote:
> Hello, freebsd-stable.
>
>  Does somebody use some software RAID5 on FreeBSD in real production
>  system?
>
>  I want to build storage server for my home: RAW photos, multi-layer
> PhotoShop files and FLAC-encoded music consume a lot of space, and
> they should be availible both from desktop & notebook.

I've used almost all combinations at one time or another.  Right now my  32
bit system with disks that are not equally sized is running gmirror/gstripe
(raid 10) and my larger 64 bit system has a 6x 750G RAIDZ1 array.

We had a discussion of this at our local geek meet last night.  The ZFS
problems seem to be mostly with complex operations (ie: databases).  I
personally havn't had a problem.  ZFS is also copy-on-write --- so snapshots
_should_ preserve original data.

My current take on home file servers is that zfs works well for those uses.
That said, my postgresql store is on ufs.

My current consulting work uses ZFS, but I tend to install opensolaris on
the fileserver as the ZFS code is fresher there.

Lev Serebryakov wrote:
>  (4) FreeBSD 7 + ZFS "zraid". And again: stability. Too many
messages
>      about locks, crashes, etc. Code is experemental. Is it only for
>      32 bit systems?
Recently I installed a new server in my local network.

[media butcher]# uname -rsm
FreeBSD 8.0-CURRENT amd64
[media butcher]# zpool list
NAME                    SIZE    USED   AVAIL    CAP  HEALTH     ALTROOT
video                  3,16T   2,01T   1,14T    63%  ONLINE     -
[media butcher]# zpool status
   pool: video
  state: ONLINE
  scrub: none requested
config:

         NAME               STATE     READ WRITE CKSUM
         video              ONLINE       0     0     0
           raidz1           ONLINE       0     0     0
             label/DISK_01  ONLINE       0     0     0
             label/DISK_02  ONLINE       0     0     0
             label/DISK_03  ONLINE       0     0     0
             label/DISK_04  ONLINE       0     0     0
             label/DISK_05  ONLINE       0     0     0
             label/DISK_06  ONLINE       0     0     0
             label/DISK_07  ONLINE       0     0     0

errors: No known data errors
[media butcher]# sysctl hw.physmem
hw.physmem: 2136186880
[media butcher]# cat /boot/loader.conf
zfs_load="YES"
vm.kmem_size="1342177280"       # 1280 MB
vm.kmem_size_max="1342177280"   # 1280 MB

So, server works very nice on moderate load with
vsftpd (with sendfile enabled) - 40..70 users online.

-- 
WBR, Andrey V. Elsukov