zfs discuss - Jul 2010 - Recommended RAM for ZFS on various platforms

I''m currently planning on running FreeBSD with ZFS, but I wanted to
double-check
how much memory I''d need for it to be stable.  The ZFS wiki currently
says you
can go as low as 1 GB, but recommends 2 GB; however, elsewhere I''ve
seen someone
claim that you need at least 4 GB.  Does anyone here know how much RAM FreeBSD 
would need in this case?

Likewise, how much RAM does OpenSolaris need for stability when running ZFS? 
 How about other OpenSolaris-based OSs, like NexentaStor?  (My searching found 
that OpenSolaris recommended at least 1 GB, while NexentaStor said 2 GB was 
okay, 4 GB was better.  I''d be interested in hearing your input,
though.)

If it matters, I''m currently planning on RAID-Z2 with 4x500GB
consumer-grade
SATA drives.  (I know that''s not a very efficient configuration, but
I''d really
like the redundancy of RAID-Z2 and I just don''t need more than 1 TB of
available
storage right now, or for the next several years.)  This is on an AMD64 system, 
and the OS in question will be running inside of VirtualBox, with raw access to 
the drives.

Thanks,
Michael

1GB isn''t enough for a real system.  2GB is a bare minimum.  If
you''re
going to use dedup, plan on a *lot* more.  I think 4 or 8 GB are good
for a typical desktop or home NAS setup.  With FreeBSD you may be able
to get away with less.  (Probably, in fact.)

Btw, instead of RAIDZ2, I''d recommend simply using stripe of mirrors.
You''ll have better performance, and good resilience against errors. 
And
you can grow later as you need to by just adding additional drive pairs.

	-- Garrett

On Fri, 2010-07-16 at 10:24 -0700, Michael Johnson
wrote:
> I''m currently planning on running FreeBSD with ZFS, but I wanted
to double-check
> how much memory I''d need for it to be stable.  The ZFS wiki
currently says you
> can go as low as 1 GB, but recommends 2 GB; however, elsewhere
I''ve seen someone
> claim that you need at least 4 GB.  Does anyone here know how much RAM
FreeBSD
> would need in this case?
> 
> Likewise, how much RAM does OpenSolaris need for stability when running
ZFS?
>  How about other OpenSolaris-based OSs, like NexentaStor?  (My searching
found
> that OpenSolaris recommended at least 1 GB, while NexentaStor said 2 GB was
> okay, 4 GB was better.  I''d be interested in hearing your input,
though.)
> 
> If it matters, I''m currently planning on RAID-Z2 with 4x500GB
consumer-grade
> SATA drives.  (I know that''s not a very efficient configuration,
but I''d really
> like the redundancy of RAID-Z2 and I just don''t need more than 1
TB of available
> storage right now, or for the next several years.)  This is on an AMD64
system,
> and the OS in question will be running inside of VirtualBox, with raw
access to
> the drives.
> 
> Thanks,
> Michael
> 
> 
>       
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

Garrett D''Amore wrote:
> Btw, instead of RAIDZ2, I''d recommend simply using stripe of
mirrors.
> You''ll have better performance, and good resilience against
errors.  And
> you can grow later as you need to by just adding additional drive pairs.
>
> 	-- Garrett
>   
Or in my case, I find my home data growth is slightly less than the rate 
of disk capacity increase, so every 18 months or so, I simply swap out 
the disks for higher capacity ones.


-- 
Andrew Gabriel

Garrett D''Amore wrote:
>On Fri, 2010-07-16 at 10:24 -0700, Michael Johnson wrote:
>> I''m currently planning on running FreeBSD with ZFS, but I
wanted to
>>double-check 
>> how much memory I''d need for it to be stable.  The ZFS wiki
currently says
>you 
>> can go as low as 1 GB, but recommends 2 GB; however, elsewhere
I''ve seen
>>someone 
>> claim that you need at least 4 GB.  Does anyone here know how much RAM 
>FreeBSD 
>> would need in this case?
>> 
>> Likewise, how much RAM does OpenSolaris need for stability when running
ZFS?
>>  How about other OpenSolaris-based OSs, like NexentaStor?  (My
searching
>found 
>> that OpenSolaris recommended at least 1 GB, while NexentaStor said 2 GB
was
>> okay, 4 GB was better.  I''d be interested in hearing your
input, though.)
>
>1GB isn''t enough for a real system.  2GB is a bare minimum.  If
you''re
>going to use dedup, plan on a *lot* more.  I think 4 or 8 GB are good
>for a typical desktop or home NAS setup.  With FreeBSD you may be able
>to get away with less.  (Probably, in fact.)
Fortunately, I don''t need deduplication; it''s kind of a nice
feature, but the
extra RAM it would take isn''t worth it.

Just curious, why do you say I''d be able to get away with less RAM in
FreeBSD
(as compared to NexentaStor, I''m assuming)?  I don''t know tons
about the OSs in
question; is FreeBSD just leaner in general?
>> If it matters, I''m currently planning on RAID-Z2 with 4x500GB
consumer-grade
>> SATA drives.  (I know that''s not a very efficient
configuration, but I''d
>>really 
>> like the redundancy of RAID-Z2 and I just don''t need more than
1 TB of
>>available 
>> storage right now, or for the next several years.)  This is on an AMD64
>>system, 
>> and the OS in question will be running inside of VirtualBox, with raw
access
>>to 
>> the drives.
>
>Btw, instead of RAIDZ2, I''d recommend simply using stripe of
mirrors.
>You''ll have better performance, and good resilience against errors.
And
>you can grow later as you need to by just adding additional drive pairs.

A pair of mirrors would be nice, but would only protect against 100% of one 
drive failing, and 50% of two-drive failures.  Performance is less important to 
me than redundancy; this setup won''t be seeing tons of disk activity,
but I want
it to be as reliable as possible.

Michael

On Fri, Jul 16, 2010 at 10:24 AM, Michael Johnson
<mjjohnson.geo at yahoo.com> wrote:
> I''m currently planning on running FreeBSD with ZFS, but I wanted
to double-check
> how much memory I''d need for it to be stable. ?The ZFS wiki
currently says you
> can go as low as 1 GB, but recommends 2 GB; however, elsewhere
I''ve seen someone
> claim that you need at least 4 GB. ?Does anyone here know how much RAM
FreeBSD
> would need in this case?
There''s no such thing as "too much RAM" when it comes to ZFS.
The
more RAM you add to the system, the better it will perform.  ZFS will
use all the RAM you give it for the ARC, enabling it to cache more and
more data.

On the flip side, if you spend enough time tuning ZFS and FreeBSD, you
can use ZFS on a system with 512 MB of RAM (there are reports on the
FreeBSD mailing lists of various people doing thing on single-drive
laptops).

However, the "rule of thumb" for ZFS is 2 GB of RAM as a bare minimum,
using the 64-bit version of FreeBSD.  The "sweet spot" is 4 GB of RAM.

But, more is always better.

-- 
Freddie Cash
fjwcash at gmail.com

On Fri, 2010-07-16 at 11:57 -0700, Michael Johnson
wrote:
> us, why do you say I''d be able to get away with less RAM in
FreeBSD
> (as compared to NexentaStor, I''m assuming)?  I don''t know
tons about
> the OSs in 
> question; is FreeBSD just leaner in general? 
Compared to Solaris, in my estimation, yes, its a little leaner.  Not
necessarily a lot -- the bulk of memory consumption these days is ZFS
and applications (Firefox!)

	- Garrett

On Fri, 16 Jul 2010, Michael Johnson wrote:
>
> Just curious, why do you say I''d be able to get away with less RAM
in FreeBSD
> (as compared to NexentaStor, I''m assuming)?  I don''t know
tons about the OSs in
> question; is FreeBSD just leaner in general?
The FreeBSD OS itself is normally leaner but FreeBSD plus zfs is not 
(yet) as memory efficient as Solaris.  Solaris and zfs do the Vulcan 
mind-meld when it comes to memory but FreeBSD does not.

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Fri, Jul 16, 2010 at 12:24 PM, Michael Johnson
<mjjohnson.geo at yahoo.com> wrote:
> I''m currently planning on running FreeBSD with ZFS, but I wanted
to double-check
> how much memory I''d need for it to be stable. ?The ZFS wiki
currently says you
> can go as low as 1 GB, but recommends 2 GB; however, elsewhere
I''ve seen someone
> claim that you need at least 4 GB. ?Does anyone here know how much RAM
FreeBSD
> would need in this case?
>
> Likewise, how much RAM does OpenSolaris need for stability when running
ZFS?
> ?How about other OpenSolaris-based OSs, like NexentaStor? ?(My searching
found
> that OpenSolaris recommended at least 1 GB, while NexentaStor said 2 GB was
> okay, 4 GB was better. ?I''d be interested in hearing your input,
though.)
>
> If it matters, I''m currently planning on RAID-Z2 with 4x500GB
consumer-grade
> SATA drives. ?(I know that''s not a very efficient configuration,
but I''d really
> like the redundancy of RAID-Z2 and I just don''t need more than 1
TB of available
> storage right now, or for the next several years.) ?This is on an AMD64
system,
> and the OS in question will be running inside of VirtualBox, with raw
access to
> the drives.
>
> Thanks,
> Michael
>
using FreeBSD 9 w/ ZFSv15 using default settings, nothing in loader.conf
or nothing in sysctl.conf and a GENERIC kernel

12GB of memory seems to be all ZFS wanted to use, I have tried
machines with 32GB
but zfs never wants to use more unless you play with loader.conf settings


if I was building a small office NAS 12GB would be where I would start.
I run mostly whitebox hardware (asus motherboards desktop disks etc..)

in my experience, I have found FreeBSD to be much more stable than Open Solaris
but to be fair, I understand FreeBSD, and I have only loaded Open solaris

with default settings and the most ram I ever gave open Solaris is a 8GB machine
so if you wanted to go with Open solaris (for dedupe and such) I would
use a Lot of ram

from many people I have talked to 32GB of memory and Open Soalris is
really stable

-- 

Sam Fourman Jr.
Fourman Networks
http://www.fourmannetworks.com

"Sam Fourman Jr." <sfourman at gmail.com> wrote:
> using FreeBSD 9 w/ ZFSv15 using default settings, nothing in loader.conf
> or nothing in sysctl.conf and a GENERIC kernel
>
> 12GB of memory seems to be all ZFS wanted to use, I have tried
> machines with 32GB
> but zfs never wants to use more unless you play with loader.conf settings
On Solaris on a SunFire X4540 with 64 GB, I''ve seen ZFS RAM usage far
beyond
32 GB without doing anything. 

Do you know whether this FreeBSD behavior is intended?



J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

On 2010-Jul-17 01:24:57 +0800, Michael Johnson <mjjohnson.geo at
yahoo.com> wrote:
>I''m currently planning on running FreeBSD with ZFS, but I wanted to
double-check
>how much memory I''d need for it to be stable.  The ZFS wiki
currently says you
>can go as low as 1 GB, but recommends 2 GB; however, elsewhere I''ve
seen someone
>claim that you need at least 4 GB.  Does anyone here know how much RAM
FreeBSD
>would need in this case?
I am running FreeBSD 8.x with ZFS on several systems.  From my
experiences, 2GB is a bare minimum and it seems a lot happier with
3.5GB.  Note that in any case, patching ARC to work around the out-
of-free-memory bug is fairly important.
>Likewise, how much RAM does OpenSolaris need for stability when running ZFS?
I have a SPARC OSol system that seemed OK with 2GB, though I never really
stres it.

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100719/d44b7e49/attachment.bin>

On Jul 18, 2010, at 3:40 PM, Peter Jeremy wrote:
> On 2010-Jul-17 01:24:57 +0800, Michael Johnson <mjjohnson.geo at
yahoo.com> wrote:
>> I''m currently planning on running FreeBSD with ZFS, but I
wanted to double-check
>> how much memory I''d need for it to be stable.  The ZFS wiki
currently says you
>> can go as low as 1 GB, but recommends 2 GB; however, elsewhere
I''ve seen someone
>> claim that you need at least 4 GB.  Does anyone here know how much RAM
FreeBSD
>> would need in this case?
> 
> I am running FreeBSD 8.x with ZFS on several systems.  From my
> experiences, 2GB is a bare minimum and it seems a lot happier with
> 3.5GB.  Note that in any case, patching ARC to work around the out-
> of-free-memory bug is fairly important.
Do you have a CR for this bug?
 -- richard

-- 
ZFS and performance consulting
http://www.RichardElling.com

On 2010-Jul-19 07:15:45 +0800, Richard Elling <richard.elling at
gmail.com> wrote:
>On Jul 18, 2010, at 3:40 PM, Peter Jeremy wrote:
>> 3.5GB.  Note that in any case, patching ARC to work around the out-
>> of-free-memory bug is fairly important.
>
>Do you have a CR for this bug?
This is a FreeBSD-specific bug related to interaction between ZFS ARC
and the FreeBSD VM - it''s not a ZFS bug.  It''s fairly clearly
described
in http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/146410 but there are
a number of other open PRs that describe similar problems.

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100719/ce38a828/attachment.bin>

> I''m currently planning on running FreeBSD with ZFS, but I wanted
to
> double-check how much memory I''d need for it to be stable. The ZFS
> wiki currently says you can go as low as 1 GB, but recommends 2 GB; 
> however, elsewhere I''ve seen someone claim that you need at least
4 GB.
> ...
> How about other OpenSolaris-based OSs, like NexentaStor?  
> ...
> If it matters, I''m currently planning on RAID-Z2 with 4x500GB 
> consumer-grade SATA drives.  ...  This is on an AMD64 system, 
> and the OS in question will be running inside of VirtualBox ...
> Thanks,
> Michael
> 
Buy the biggest Chips you can afford and if you need to pair them (for
performance)
do so. You want to keep as many Memory Slots open as you can so you can add more
memory later. I think you (or I) would be unhappy with a measly 4GB in a new
System
but in reality it would be OK.

If it is not OK (for you) then you have open Memory Slots in which to add more
Chips (which you are certain to want to do in the future).

Rob
-- 
This message posted from opensolaris.org