zfs discuss - Nov 2008 - Still more questions WRT selecting a mobo for small ZFS RAID

Like many others, I am looking to put together a SOHO NAS based on ZFS/CIFS. 
The plan is 6 x 1TB drives in RAIDZ2 configuration, driven via mobo with 6 SATA
ports.

I''ve read most, if not all, of the threads here, as well as
sbredon''s excellent article on building a home NAS, yet I still have a
number of unanswered questions.

I was leaning heavily towards the M2N-E for a while, but it has a number of
strikes against it:

   - Discontinued, not likely to be available to build additional rigs
   - No longer available via Newegg
   - MCP55 copy/fs lockup problem still unresolved

Although the Nvidia 750a chipset seems to be supported by the sata_nv driver, I
have read several open bug reports which indicate it may not be a good choice. 
Likewise, I have read several negatives on the entire line of AMD southbridges
under Solaris. As it stands, the only compelling reason to go with an AMD CPU
seems to be support for ECC in inexpensive motherboards.

On the Intel side of things, there seem to be plenty of good reports of the ICH
7/8/9 chipsets that support AHCI and no negatives.  The only negative on the
Intel side seems to be the lack of ECC memory outside of expensive server mobos.

As things stand, I am leaning towards a P35/ICH9 or P45/ICH10 setup.  In
particular, I am considering either the P5E-VM HDMI for a small integrated
solution or the GA-EP45-UD3P for something with greater expandability.

So, my questions:

- Has the MCP55 copy/fs lockup bug been fixed yet?

- Have the Nvidia 750a driver issues been resolved?

- Are the AMD southbridges still in the not-recomemded category?

- Are there any downsides to going with with an ICH controller with the AHCI
driver?

- Are there any performance issues to be aware of with the ICH vs the MCP55?

- Are there any inexpensive Intel mobos that do support ECC memory and the
latest Intel 45 nm CPUs?

- Any comments on the P5E-VM or GA-EP45 boards?

Many thanks in advance!
-- 
This message posted from opensolaris.org

gnomad wrote:
> So, my questions:
>
> - Has the MCP55 copy/fs lockup bug been fixed yet?
>
>   
Which bug ids? I''ve never seen any such problems in 18 months of heavy
use.  Note the x4540 uses these.
> - Have the Nvidia 750a driver issues been resolved?
>
>   
Which bug ids?

-- 
Ian.

On Fri, Nov 14, 2008 at 4:43 PM, gnomad <gnomad at gmail.com>
wrote:
> Like many others, I am looking to put together a SOHO NAS based on
ZFS/CIFS.  The plan is 6 x 1TB drives in RAIDZ2 configuration, driven via mobo
with 6 SATA ports.
>
> I''ve read most, if not all, of the threads here, as well as
sbredon''s excellent article on building a home NAS, yet I still have a
number of unanswered questions.
>
> I was leaning heavily towards the M2N-E for a while, but it has a number of
strikes against it:
>
>   - Discontinued, not likely to be available to build additional rigs
>   - No longer available via Newegg
>   - MCP55 copy/fs lockup problem still unresolved
>
> Although the Nvidia 750a chipset seems to be supported by the sata_nv
driver, I have read several open bug reports which indicate it may not be a good
choice.  Likewise, I have read several negatives on the entire line of AMD
southbridges under Solaris. As it stands, the only compelling reason to go with
an AMD CPU seems to be support for ECC in inexpensive motherboards.
>
> On the Intel side of things, there seem to be plenty of good reports of the
ICH 7/8/9 chipsets that support AHCI and no negatives.  The only negative on the
Intel side seems to be the lack of ECC memory outside of expensive server mobos.
>
> As things stand, I am leaning towards a P35/ICH9 or P45/ICH10 setup.  In
particular, I am considering either the P5E-VM HDMI for a small integrated
solution or the GA-EP45-UD3P for something with greater expandability.
>
> So, my questions:
>
> - Has the MCP55 copy/fs lockup bug been fixed yet?
>
> - Have the Nvidia 750a driver issues been resolved?
>
> - Are the AMD southbridges still in the not-recomemded category?
>
> - Are there any downsides to going with with an ICH controller with the
AHCI driver?
>
> - Are there any performance issues to be aware of with the ICH vs the
MCP55?
>
> - Are there any inexpensive Intel mobos that do support ECC memory and the
latest Intel 45 nm CPUs?
>
> - Any comments on the P5E-VM or GA-EP45 boards?
>
> Many thanks in advance!
> --
I''m going to be somewhat rude and bypass your list of detailed
questions - but give you my thoughts on a motherboard recommendation
(and other hardware).

a) related to the 1Tb disks, I''d highly recommend the WD Caviar Black
drive.  Its fast and the firmware does a great job on different
workloads that vary between large file sequencial read (workloads) to
(workloads that demand) lots of small random reads/writes.  Their
"dual processor" controller architecture really works.

b) If I were building a system today, I''d go Intel -  even thought
I''m
an AMD fanboy - but I can''t recommend AMD today ... unfortunately.

c) RAM is the most important attribute of a ZFS based server.  Think
lots of RAM.  Unfortunately, Intel has turned the market into a
two-tier market, with the lower (price) tier limited to 4 DIMM slots.
So, pick a board that has been tested with 4 * 2Gb or 4 * 4Gb DIMM
configs and plan on building a system with at least 4*2Gb DIMMs today.

c1) If you have a choice, based on your budgetary constraints, between
(for example) 4*1G of "performance" RAM and 4*2Gb of "value"
(main
stream performance) RAM - go with value RAM.  Whatever you do,  PLEASE
maximize system memory capacity.

d) The P45 based boards are a no-brainer.  Great performance, good
pricing, reasonable power consumption and highly mature.

e) If the board is going to be *only* used as a NAS, the current CPU
"sweet spot" is, IMHO, the Intel 	Intel Core 2 Duo E7200 (45nm, 2.53
GHz, 3MB L2 Cache).  Plenty of "horsepower", low-power consumption,
nice cache capacity and priced to go!

f) If you intend to use the box for other demanding tasks (for
example, running other OS under VirtualBox) and need more CPU power,
I''d pick the E8400 (dual core).   But remember, the priority is RAM
capacity first, upgraded CPU second.  I really think that the E7200
will work well in your application.

f) Look at the following reviews and pick a motherboard from any of them:

http://www.tomshardware.com/reviews/energy-efficient-motherboard,2067.html
http://techreport.com/articles.x/15737
http://www.tomshardware.com/reviews/intel-p45-motherboard,2001.html

I really don''t think you can go wrong with any Intel based system that
has had a halfway decent review report card.

There has *never* been a better time to buy/build a system.

HTH.  Email me offlist if I can help.

Regards,

-- 
Al Hopper  Logical Approach Inc,Plano,TX al at logical-approach.com
                   Voice: 972.379.2133 Timezone: US CDT
OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/

I looked at this a month back, i was leaning towards intel for  
performance and power consumption but went for AMD doe to lack of ECC  
support in most of the Intel chipsets.

I went for a AM2+ GeForce 8200 motherboard which seemed more stable  
with Solaris than 8300. With the AM2+ socket I can wait for the new  
45nm CPUs, I bought the cheapest dual-core I could find for now (which  
did not support PM). I am very happy with the system except for the  
fact that the onboard NIC doesn''t work.

On Nov 14, 2008, at 11:43 PM, gnomad wrote:
>
>
> - Any comments on the P5E-VM or GA-EP45 boards?
Nether of them support ECC-memory.
>
>
> Many thanks in advance!
> -- 
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Henrik Johansson
http://sparcv9.blogspot.com

On Fri, 14 Nov 2008, Al Hopper wrote:
>
> b) If I were building a system today, I''d go Intel -  even thought
I''m
> an AMD fanboy - but I can''t recommend AMD today ... unfortunately.
Is there some particular reason for this?  The now shipping 0.45 
micron quad-core Opterons seem quite nice indeed.  As far as I am 
aware, Opteron still has an advantage over Intel with moving data on 
and off the CPU.

Bob
=====================================Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Fri, Nov 14, 2008 at 7:12 PM, Bob Friesenhahn
<bfriesen at simple.dallas.tx.us> wrote:
> On Fri, 14 Nov 2008, Al Hopper wrote:
>>
>> b) If I were building a system today, I''d go Intel -  even
thought I''m
>> an AMD fanboy - but I can''t recommend AMD today ...
unfortunately.
>
> Is there some particular reason for this?  The now shipping 0.45 micron
> quad-core Opterons seem quite nice indeed.  As far as I am aware, Opteron
> still has an advantage over Intel with moving data on and off the CPU.
>
The just released 45nm AMD parts[0] are intended for Socket F
(formerly occupied by Opteron parts) - which is part of the 2nd (more
expensive) server (pricing) tier.  And these parts are priced
accordingly.  In a few months, AMD will release the workstation (lower
priced tier) version of the current Shanghai (aka Phenom II) part -
and it''s likely that this part will be absolutely killed by the
equivalent version of the Nahelem (aka Core i7) parts that will be
available at the same time.  Actually the Core i7 parts will probably
be widely available before the AMD equivalent parts. :(

So #1:  I said "now" (today)[1]  and #2 the motherboards and CPUs I
mentioned will allow the OP to build a system where the overall
performance will be limited by the system disk drives and installed
memory.  Building a system with more  internal memory to CPU
bandwidth, or some other exotic system feature, will not change that.

Another point worth mentioning is that its far better, and economical,
to build a home NAS based on proven, mature, widely available
commodity parts rather than trying to build it based on brand new, rev
~ 1.0 BIOS based products that have not even been reviewed by the
usual tech review sites.[2]

[0] http://it.anandtech.com/IT/showdoc.aspx?i=3456
[1] Neither Core i7 parts or AMD 45nm Shanghai parts are listed on newegg.com.
[2] nothing on tomshardware.com or many other review sites in terms of
detailed performance level evaluations of the AMD Shanghai parts.

Regards,

-- 
Al Hopper  Logical Approach Inc,Plano,TX al at logical-approach.com
                   Voice: 972.379.2133 Timezone: US CDT
OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/

> WD Caviar Black drive [...] Intel E7200 2.53GHz 3MB L2
 > The P45 based boards are a no-brainer

16G of DDR2-1066 with P45 or
  8G of ECC DDR2-800 with 3210 based boards

That is the question.

		Rob

Rob at Logan.com wrote:
>  > WD Caviar Black drive [...] Intel E7200 2.53GHz 3MB L2
>  > The P45 based boards are a no-brainer
>
> 16G of DDR2-1066 with P45 or
>   8G of ECC DDR2-800 with 3210 based boards
>
> That is the question.
>
>   
I guess the answer is how valuable is your data?

-- 
Ian.

>I looked at this a month back, i was leaning towards intel for  
>performance and power consumption but went for AMD doe to lack of ECC  
>support in most of the Intel chipsets.
>
>I went for a AM2+ GeForce 8200 motherboard which seemed more stable  
>with Solaris than 8300. With the AM2+ socket I can wait for the new  
>45nm CPUs, I bought the cheapest dual-core I could find for now (which  
>did not support PM). I am very happy with the system except for the  
>fact that the onboard NIC doesn''t work.
Which NIC is that?

Casper

On Nov 15, 2008, at 11:18 AM, Casper.Dik at Sun.COM wrote:
>>
>>
>> I went for a AM2+ GeForce 8200 motherboard which seemed more stable
>> with Solaris than 8300. With the AM2+ socket I can wait for the new
>> 45nm CPUs, I bought the cheapest dual-core I could find for now  
>> (which
>> did not support PM). I am very happy with the system except for the
>> fact that the onboard NIC doesn''t work.
>
> Which NIC is that?
NVIDIA nForce built-in Gigabit MAC with external RTL8211CL-GR (ASUS  
M3N78-AM), both the HCL and some posts in forums stated that it did  
not work, so I bought an Intel card for it also.

Henrik Johansson
http://sparcv9.blogspot.com

On Sat, 15 Nov 2008 18:49:17 +1300
Ian Collins <ian at ianshome.com> wrote:
> Rob at Logan.com wrote:
> >  > WD Caviar Black drive [...] Intel E7200 2.53GHz 3MB L2
> >  > The P45 based boards are a no-brainer
> >
> > 16G of DDR2-1066 with P45 or
> >   8G of ECC DDR2-800 with 3210 based boards
> >
> > That is the question.
> >   
> I guess the answer is how valuable is your data?
I disagree. The answer is: go for the 16G and make backups. The 16G
system will work far more "easy" and I may be lucky but in the past
years I did not have ZFS issues with my non-ECC ram ;-)

-- 
Dick Hoogendijk -- PGP/GnuPG key: 01D2433D
+ http://nagual.nl/ | SunOS sxce snv101 ++
+ All that''s really worth doing is what we do for others (Lewis Carrol)

dick hoogendijk wrote:
> On Sat, 15 Nov 2008 18:49:17 +1300
> Ian Collins <ian at ianshome.com> wrote:
>
>   
>> Rob at Logan.com wrote:
>>     
>>>  > WD Caviar Black drive [...] Intel E7200 2.53GHz 3MB L2
>>>  > The P45 based boards are a no-brainer
>>>
>>> 16G of DDR2-1066 with P45 or
>>>   8G of ECC DDR2-800 with 3210 based boards
>>>
>>> That is the question.
>>>   
>>>       
>> I guess the answer is how valuable is your data?
>>     
>
> I disagree. The answer is: go for the 16G and make backups. The 16G
> system will work far more "easy" and I may be lucky but in the
past
> years I did not have ZFS issues with my non-ECC ram ;-)
>   
You are lucky.  I recommend ECC RAM for any data that you care
about.  Remember, if there is a main memory corruption, that may
impact the data that ZFS writes which will negate any on-disk
redundancy.  And yes, this does occur -- check the archives for the
tales of woe.
 -- richard

Henrik Johansson wrote:
> On Nov 15, 2008, at 11:18 AM, Casper.Dik at Sun.COM wrote:
>
>   
>>> I went for a AM2+ GeForce 8200 motherboard which seemed more stable
>>> with Solaris than 8300. With the AM2+ socket I can wait for the new
>>> 45nm CPUs, I bought the cheapest dual-core I could find for now  
>>> (which
>>> did not support PM). I am very happy with the system except for the
>>> fact that the onboard NIC doesn''t work.
>>>       
>> Which NIC is that?
>>     
>
> NVIDIA nForce built-in Gigabit MAC with external RTL8211CL-GR (ASUS  
> M3N78-AM), both the HCL and some posts in forums stated that it did  
> not work, so I bought an Intel card for it also.
>   
RTL8211C IP checksum offload is broken.  You can disable it, but you
have to edit /etc/system.  See CR 6686415 for details.
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6686415
 -- richard

dick hoogendijk wrote:
 > > Ian Collins <ian at ianshome.com> wrote:
 >> >> Rob at Logan.com wrote:
 >> >>
 >>> >>>  > WD Caviar Black drive [...] Intel E7200 2.53GHz
3MB L2
 >>> >>>  > The P45 based boards are a no-brainer
 >>> >>>
 >>> >>> 16G of DDR2-1066 with P45 or
 >>> >>>  8G of ECC DDR2-800 with 3210 based boards
 >>> >>>
 >>> >>> That is the question.
 >>> >>>
 >> >> I guess the answer is how valuable is your data?
 >> >>
 > > I disagree. The answer is: go for the 16G and make backups.

getting corruption passing through zil onto disk is a risk,
perhaps smaller if metadata, (sole risk to pool integrity) but random system
crashes and crupt answers from the arc cache should be expected with 16G
filled for months. (still, might not be an issue for a single home user,
but if your married it might be :-)

the Enterprise version of the above drive is
http://www.wdc.com/en/products/Products.asp?DriveID=503
possibly with a desirable faster timeout.

			Rob

>Henrik Johansson wrote:
>> On Nov 15, 2008, at 11:18 AM, Casper.Dik at Sun.COM wrote:
>>
>>   
>>>> I went for a AM2+ GeForce 8200 motherboard which seemed more
stable
>>>> with Solaris than 8300. With the AM2+ socket I can wait for the
new
>>>> 45nm CPUs, I bought the cheapest dual-core I could find for now
>>>> (which
>>>> did not support PM). I am very happy with the system except for
the
>>>> fact that the onboard NIC doesn''t work.
>>>>       
>>> Which NIC is that?
>>>     
>>
>> NVIDIA nForce built-in Gigabit MAC with external RTL8211CL-GR (ASUS  
>> M3N78-AM), both the HCL and some posts in forums stated that it did  
>> not work, so I bought an Intel card for it also.
>>   
>
>RTL8211C IP checksum offload is broken.  You can disable it, but you
>have to edit /etc/system.  See CR 6686415 for details.
>http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6686415
> -- richard

I think the proper way to state this is "the driver doesn''t
properly
support checksum offload".  (In many of the newer realtek cards the
way the offload is done is differently)

Casper

On Sat, Nov 15, 2008 at 9:26 AM, Richard Elling <Richard.Elling at
sun.com> wrote:
> dick hoogendijk wrote:
>> On Sat, 15 Nov 2008 18:49:17 +1300
>> Ian Collins <ian at ianshome.com> wrote:
>>
>>
>>> Rob at Logan.com wrote:
>>>
>>>>  > WD Caviar Black drive [...] Intel E7200 2.53GHz 3MB L2
>>>>  > The P45 based boards are a no-brainer
>>>>
>>>> 16G of DDR2-1066 with P45 or
>>>>   8G of ECC DDR2-800 with 3210 based boards
>>>>
>>>> That is the question.
>>>>
>>>>
>>> I guess the answer is how valuable is your data?
>>>
>>
>> I disagree. The answer is: go for the 16G and make backups. The 16G
>> system will work far more "easy" and I may be lucky but in
the past
>> years I did not have ZFS issues with my non-ECC ram ;-)
>>
>
> You are lucky.  I recommend ECC RAM for any data that you care
> about.  Remember, if there is a main memory corruption, that may
> impact the data that ZFS writes which will negate any on-disk
> redundancy.  And yes, this does occur -- check the archives for the
> tales of woe.
I agree with your recommendation Richard.  OTOH I''ve built/used a
bunch of systems over several years that were mostly non ECC equipped
and only lost one DIMM along the way.  So I guess I''ve been lucky also
- but IMHO the failure rate for RAM these days is pretty small[1].
I''ve also been around hundreds of SPARC boxes and, again, very, few
RAM failures (one is all that I can remember).

Risk management is exactly that.  You have to determine where the risk
is and how important it is and how likely it is to bite.  And then
allocate costs from your budget to minimize that risk.  Remember that
you won''t totally eliminate all risk - but you can minimize it.  At
the time when there was a big cost delta between ECC and non ECC RAM
parts, I always went with the most (non ECC) RAM that the budget would
support.  That was my personal risk assessment and priority.  I think
it was a good decision and it did''nt cause me any grief.

[1] I do recommend that you test the heck out of new RAM parts and
ensure that they get some airflow - especially if they are getting a
supply of hot air from any nearby CPU coolers.  Even the simple
"finger test" will tell you if you need a fan for your RAM DIMMs.

-- 
Al Hopper  Logical Approach Inc,Plano,TX al at logical-approach.com
                   Voice: 972.379.2133 Timezone: US CDT
OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/

Al Hopper wrote:
> On Sat, Nov 15, 2008 at 9:26 AM, Richard Elling <Richard.Elling at
sun.com> wrote:
>   
>> dick hoogendijk wrote:
>>     
>>> On Sat, 15 Nov 2008 18:49:17 +1300
>>> Ian Collins <ian at ianshome.com> wrote:
>>>
>>>
>>>       
>>>> Rob at Logan.com wrote:
>>>>
>>>>         
>>>>>  > WD Caviar Black drive [...] Intel E7200 2.53GHz 3MB
L2
>>>>>  > The P45 based boards are a no-brainer
>>>>>
>>>>> 16G of DDR2-1066 with P45 or
>>>>>   8G of ECC DDR2-800 with 3210 based boards
>>>>>
>>>>> That is the question.
>>>>>
>>>>>
>>>>>           
>>>> I guess the answer is how valuable is your data?
>>>>
>>>>         
>>> I disagree. The answer is: go for the 16G and make backups. The 16G
>>> system will work far more "easy" and I may be lucky but
in the past
>>> years I did not have ZFS issues with my non-ECC ram ;-)
>>>
>>>       
>> You are lucky.  I recommend ECC RAM for any data that you care
>> about.  Remember, if there is a main memory corruption, that may
>> impact the data that ZFS writes which will negate any on-disk
>> redundancy.  And yes, this does occur -- check the archives for the
>> tales of woe.
>>     
>
> I agree with your recommendation Richard.  OTOH I''ve built/used a
> bunch of systems over several years that were mostly non ECC equipped
> and only lost one DIMM along the way.  So I guess I''ve been lucky
also
> - but IMHO the failure rate for RAM these days is pretty small[1].
> I''ve also been around hundreds of SPARC boxes and, again, very,
few
> RAM failures (one is all that I can remember).
>
>   
I think the situation will change with the current expansion in RAM
sizes.  Five years ago with mainly 32 bit x86 systems, 4G of ram was a
lot (even on most Sparc boxes).  Today 32 and 64GB are becoming common. 
Desktop systems have seen similar growth.

ZFS also uses system RAM in a way it hasn''t been used before.  Memory
that would have been unused or holding static pages is now churning
rapidly, in a way similar memory testers like memtest86. Random patterns
are cycling though RAM like never before, greatly increasing the chances
for hitting a bad bit or addressing error.  I''ve had RAM faults that
have taken hours with memtest86 to hit the trigger bit pattern that
would have gone unnoticed for years if I hadn''t seen data corruption
with ZFS.

ZFS may turn out to be the ultimate RAM soak tester!

-- 
Ian.

On Sat, 15 Nov 2008 13:38:53 -0600
"Al Hopper" <al at logical-approach.com> wrote:
>  So I guess I''ve been lucky also
> - but IMHO the failure rate for RAM these days is pretty small[1].
> I''ve also been around hundreds of SPARC boxes and, again, very,
few
> RAM failures (one is all that I can remember).
> 
> Risk management is exactly that.  You have to determine where the risk
> is and how important it is and how likely it is to bite.  And then
> allocate costs from your budget to minimize that risk.
So I guess, I do have to go for ECC ram when I build a new server.
I also understood that like you wrote "The P45 based boards are a
no-brainer" Intel MoBo''s are a no-go when you want ECC ram -and-
want
it a little cheap.

So, what -is- a really good MB that supports ECC ram (min.8MB) and what
processor is recommended?

-- 
Dick Hoogendijk -- PGP/GnuPG key: 01D2433D
+ http://nagual.nl/ | SunOS sxce snv101 ++
+ All that''s really worth doing is what we do for others (Lewis Carrol)

Ian Collins wrote:
> Al Hopper wrote:
>   
>> On Sat, Nov 15, 2008 at 9:26 AM, Richard Elling <Richard.Elling at
sun.com> wrote:
>>   
>>     
>>> dick hoogendijk wrote:
>>>     
>>>       
>>>> On Sat, 15 Nov 2008 18:49:17 +1300
>>>> Ian Collins <ian at ianshome.com> wrote:
>>>>
>>>>
>>>>       
>>>>         
>>>>> Rob at Logan.com wrote:
>>>>>
>>>>>         
>>>>>           
>>>>>>  > WD Caviar Black drive [...] Intel E7200 2.53GHz
3MB L2
>>>>>>  > The P45 based boards are a no-brainer
>>>>>>
>>>>>> 16G of DDR2-1066 with P45 or
>>>>>>   8G of ECC DDR2-800 with 3210 based boards
>>>>>>
>>>>>> That is the question.
>>>>>>
>>>>>>
>>>>>>           
>>>>>>             
>>>>> I guess the answer is how valuable is your data?
>>>>>
>>>>>         
>>>>>           
>>>> I disagree. The answer is: go for the 16G and make backups. The
16G
>>>> system will work far more "easy" and I may be lucky
but in the past
>>>> years I did not have ZFS issues with my non-ECC ram ;-)
>>>>
>>>>       
>>>>         
>>> You are lucky.  I recommend ECC RAM for any data that you care
>>> about.  Remember, if there is a main memory corruption, that may
>>> impact the data that ZFS writes which will negate any on-disk
>>> redundancy.  And yes, this does occur -- check the archives for the
>>> tales of woe.
>>>     
>>>       
>> I agree with your recommendation Richard.  OTOH I''ve
built/used a
>> bunch of systems over several years that were mostly non ECC equipped
>> and only lost one DIMM along the way.  So I guess I''ve been
lucky also
>> - but IMHO the failure rate for RAM these days is pretty small[1].
>> I''ve also been around hundreds of SPARC boxes and, again,
very, few
>> RAM failures (one is all that I can remember).
>>
>>   
>>     
> I think the situation will change with the current expansion in RAM
> sizes.  Five years ago with mainly 32 bit x86 systems, 4G of ram was a
> lot (even on most Sparc boxes).  Today 32 and 64GB are becoming common. 
> Desktop systems have seen similar growth.
>   
Let''s do some math.  A generally accepted Soft Error Rate (SER) for 
DRAMs is
1,000 FITs or an Annualized Failure Rate (AFR) of 0.88%.  If a non-ECC DIMM
has 8 chips then your AFR is 7%, or 14% for 16 chip DIMMs.  My desktop
has 4 DIMMs at 16-chips each, so I should expect an AFR of 56%.  Since these
are soft errors, a RAM test program may not detect it.

ECC will dramatically reduce the system-level effects of SERs.  Extended ECC
will further reduce this by about 2 orders of magnitude.
> ZFS also uses system RAM in a way it hasn''t been used before. 
Memory
> that would have been unused or holding static pages is now churning
> rapidly, in a way similar memory testers like memtest86. Random patterns
> are cycling though RAM like never before, greatly increasing the chances
> for hitting a bad bit or addressing error.  I''ve had RAM faults
that
> have taken hours with memtest86 to hit the trigger bit pattern that
> would have gone unnoticed for years if I hadn''t seen data
corruption
> with ZFS.
>
> ZFS may turn out to be the ultimate RAM soak tester!
>   
:-)  no, not really.  SERs are more of a problem for idle DRAM because the
probability of a SER affecting you is a function of the time the data 
has been
sitting in RAM waiting to be affected by upsets.

Note: there are some studies suggesting a correlation between SERs and
hard faults.  In practice, it doesn''t really matter why or how the
fault
occurred, the solution is ECC, Extended ECC, or memory mirroring.
 -- richard

Casper.Dik at Sun.COM wrote:
>> RTL8211C IP checksum offload is broken.  You can disable it, but you
>> have to edit /etc/system.  See CR 6686415 for details.
>> http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6686415
>> -- richard
>>     
>
>
> I think the proper way to state this is "the driver doesn''t
properly
> support checksum offload".  (In many of the newer realtek cards the
> way the offload is done is differently)
>   
Yes, this is a better way to say it.  The bug is in, or can be solved 
by, the driver.
 -- richard

Al Hopper writes:
> I''m going to be somewhat rude and bypass your list of
> detailed
> questions - but give you my thoughts on a motherboard
> recommendation
> (and other hardware).
No worries, you''ve pretty much confirmed things I already knew.  ;-)
> a) related to the 1Tb disks, I''d highly recommend the
> WD Caviar Black
> drive.  Its fast and the firmware does a great job on
> different
> workloads that vary between large file sequencial
> read (workloads) to
> (workloads that demand) lots of small random
> reads/writes.  Their
> "dual processor" controller architecture really
> works.
I have been a fan of Seagate for the past few years, but it seems as if they
have taken a big dive in the past six months.  I was planing to go with WD for
this project, though probably Green over Black, as heat and noise are the
primary concerns.
> b) If I were building a system today, I''d go Intel -
>  even thought I''m
> n AMD fanboy - but I can''t recommend AMD today ...
> unfortunately.
Aside from the ECC issue, of course.
> c) RAM is the most important attribute of a ZFS based
> server.  Think
> lots of RAM.  Unfortunately, Intel has turned the
> market into a
> two-tier market, with the lower (price) tier limited
> to 4 DIMM slots.
> So, pick a board that has been tested with 4 * 2Gb or
> 4 * 4Gb DIMM
> configs and plan on building a system with at least
> 4*2Gb DIMMs today.
> 
> c1) If you have a choice, based on your budgetary
> constraints, between
> (for example) 4*1G of "performance" RAM and 4*2Gb of
> "value" (main
> stream performance) RAM - go with value RAM.
>  Whatever you do,  PLEASE
> aximize system memory capacity.
I was planning to go with 2 x 2G sticks (total 4G) in a four slot mobo which
would allow me to upgrade to 8G if necessary.  I think 4G should be sufficient
as I will be the only user for now.
> d) The P45 based boards are a no-brainer.  Great
> performance, good
> pricing, reasonable power consumption and highly
> mature.
While I would agree the P45 is mature in terms of mobo support, I have not seen
indication that those motherboards are mature in terms of Solaris support.
> e) If the board is going to be *only* used as a NAS,
> the current CPU
> "sweet spot" is, IMHO, the Intel 	Intel Core 2 Duo
> E7200 (45nm, 2.53
> GHz, 3MB L2 Cache).  Plenty of "horsepower",
> low-power consumption,
> nice cache capacity and priced to go!
I was actually thinking the E5200 which seems nearly as powerful at 2/3 the
price.
> f) If you intend to use the box for other demanding
> tasks (for
> example, running other OS under VirtualBox) and need
> more CPU power,
> I''d pick the E8400 (dual core).   But remember, the
> priority is RAM
> capacity first, upgraded CPU second.  I really think
> that the E7200
> will work well in your application.
And I am planning to select a mobo that will take the latest 45 nm quad cores
should I decide to do that upgrade down the road.
> I really don''t think you can go wrong with any Intel
> based system that
> has had a halfway decent review report card.
The real question is how solid the P45/ICH10 support is with Solaris, and
whether the lack of ECC supports negates much of the advantages of the
P45/Core2.  I have nothing against AMD (I was an AMD guy prior to the Core
architecture) but I just have not seen much in the way of solid reports from the
AMD mobo chipsets currently in production.

-g.
-- 
This message posted from opensolaris.org

Henrik Johansson writes:
> I looked at this a month back, i was leaning towards
> intel for  
> performance and power consumption but went for AMD
> doe to lack of ECC  
> support in most of the Intel chipsets.
This seems to be the crux of my indecision as well.
> I went for a AM2+ GeForce 8200 motherboard which
> seemed more stable  
> with Solaris than 8300.
The problem I have been having is that the best I can hear of the in-production
AMD hardware is that some is simply "more stable" than others.

-g.
-- 
This message posted from opensolaris.org

On Mon 17/11/08 09:17 , Richard Elling Richard.Elling at Sun.COM
sent:
> Ian Collins wrote:
> 
> > ZFS also uses system RAM in a way it hasn''t been
> used before.  Memory> that would have been unused or holding static
> pages is now churning> rapidly, in a way similar memory testers like
> memtest86. Random patterns> are cycling though RAM like never before,
> greatly increasing the chances> for hitting a bad bit or addressing
error.  I''ve
> had RAM faults that> have taken hours with memtest86 to hit the
> trigger bit pattern that> would have gone unnoticed for years if I
hadn''t
> seen data corruption> with ZFS.
> >
> > ZFS may turn out to be the ultimate RAM soak
> tester!>   
> 
> :-)  no, not really.  SERs are more of a problem for idle DRAM because
> theprobability of a SER affecting you is a function of the time the data 
> has been
> sitting in RAM waiting to be affected by upsets.
> 
Maybe not for soft errors, but more so for hard errors.  The last faulty DIMM I
had had been in use for more than a year before I started using ZFS on that
system.  Within a few days I had I/O errors reported by ZFS.

It may have been a coincidence, but I don''t believe in those!

-- 
Ian.

On Sun, 16 Nov 2008, Richard Elling wrote:
>
> Let''s do some math.  A generally accepted Soft Error Rate (SER)
for
> DRAMs is
> 1,000 FITs or an Annualized Failure Rate (AFR) of 0.88%.  If a non-ECC DIMM
> has 8 chips then your AFR is 7%, or 14% for 16 chip DIMMs.  My desktop
> has 4 DIMMs at 16-chips each, so I should expect an AFR of 56%.  Since
these
> are soft errors, a RAM test program may not detect it.
This does not consider the possibility of a motherboard problem.  In 
my case, a partial motherboard failure caused many ECC events.  It was 
as if a couple of DIMMs were failing.  Solaris/ECC did the right thing 
to isolate the failing parts so I was not aware of the problem at all 
except for a fault report.

Bob
=====================================Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/