zfs discuss - Nov 2009 - The 100,000th beginner question about a zfs server

With apologies for clogging up the forum with beginner questions - 

I''m trying to figure out how to build a home zfs server. Common
question. In the last two months of reading the net and here, I''ve
found many answers, none of which would convince me to part with the $800-$1K to
do it.

So can someone take pity on a beginner and tell me - will this work?
It''s about my seventeenth paper design for the server. Objectives are
1 - zfs and other considerations for data preservation
2 - file server operation
3 - units of terabytes, consistent with a few (less than a dozen) disks
4 - as low an electrical power use as practical, given the above

I''ve thrashed through Intel vs AMD, ECC, chipset support, number of
ports, adapters, and so on ad nauseum. Here''s what I think will work:

Supermicro MBD-X7SBL-LN1-O
intel Xeon E3110
unregistered ECC, 4GB - 8GB

What I can''t pick out of the overwhelming flood of raw data
I''ve read is:
1 - does opensolaris offer driver support for the SATA ports resident on the
motherboard (Intel? 3200 + ICH9R), or must I get another board to run them?
I''m happy with the six SATA ports on the MB to start with
2 - does opensolaris directly support the LAN chips ( Intel 82573V) on that MB,
or must I grab a NIC to stick in a slot?
3 - does opensolaris support that graphics chipset (XGI Volari Z9S) well enough
to let me install and get it bootstrapped into operation, after which
I''ll make it headless.
4 - Are there any gotchas which would keep me from enabling and running the ECC
memory functions productively; I think this is a "no", but as long as
I''m asking questions...

Things of some considerations but lesser importance:
5. Electrical power; the E3110 is a low-ish power chip (nominally 65W) Low power
is nice, but not a killer. I''d prefer it to be low, and am willing to
take slow to get more of that, because my file server needs are not in any way
real time. I just need a large, but reliable, bit bucket.
6. Cost; I''ve been through several iterations of something with an AMD
Athlon 11 X2 240e with an Asus motherboard to get lower power and cost for the
same objectives. I can''t tell that the silly thing would or would not
be something I could make run. And frankly, the data is worth more than the
extra $200 or so for the intel solution - iff the intel solution works. But I
really am not interested in buying a canned commercial solution for a couple of
$K. I''m willing to put in the work setting up and managing the system
in lieu of that, so there is a dollar threshold, I guess.

And the last silly question. It seems to me that you''d have many, many
adopters if there was a real answer to what the HCL tries to be and
isn''t - an answer to "if I buy this stuff, do I have a prayer of
making it work, or is there a subtle gotcha that''s going to waste my
time and money?" We used to solve that with reference designs. They
don''t have to be perfect, they don''t have to be optimal, but
they should be practical and they should be modestly predictable given moderate
skill in the art.

I think that an intel and an AMD reference design would be a screaming good idea
for improving the acceptance and population of opensolaris.
-- 
This message posted from opensolaris.org

---------- Forwarded message ----------
From: Al Hopper <al at logical-approach.com>
Date: Sat, Nov 21, 2009 at 8:23 PM
Subject: Re: [zfs-discuss] The 100,000th beginner question about a zfs server
To: "R.G. Keen" <keen at geofex.com>


On Sat, Nov 21, 2009 at 7:08 PM, R.G. Keen <keen at geofex.com>
wrote:
>
> With apologies for clogging up the forum with beginner questions -
No problem - we''ve all been there.
> I''m trying to figure out how to build a home zfs server. Common
question. In the last two months of reading the net and here, I''ve
found many answers, none of which would convince me to part with the $800-$1K to
do it.
>
> So can someone take pity on a beginner and tell me - will this work?
It''s about my seventeenth paper design for the server. Objectives are
You get extra points for putting in this amount of effort before this post!
> 1 - zfs and other considerations for data preservation
It would be helpful to give us a broad description of what type of
data you''re planning on storing. ?Small files, large files, required
capactity etc. ?and we can probably make some specific
recommendations.
> 2 - file server operation
> 3 - units of terabytes, consistent with a few (less than a dozen) disks
> 4 - as low an electrical power use as practical, given the above
>
> I''ve thrashed through Intel vs AMD, ECC, chipset support, number
of ports, adapters, and so on ad nauseum. Here''s what I think will
work:
>
> Supermicro MBD-X7SBL-LN1-O
> intel Xeon E3110
> unregistered ECC, 4GB - 8GB
I see that your research has shown the importance of using ECC memory.
?Hence the Xeon based choice. ?Good choice.
> What I can''t pick out of the overwhelming flood of raw data
I''ve read is:
> 1 - does opensolaris offer driver support for the SATA ports resident on
the motherboard (Intel? 3200 + ICH9R), or must I get another board to run them?
I''m happy with the six SATA ports on the MB to start with
Yes the Intel ICH9R is supported.
> 2 - does opensolaris directly support the LAN chips ( Intel 82573V) on that
MB, or must I grab a NIC to stick in a slot?
The Intel based NICs are well supported by OpenSolaris and have been
historically. ?Good choice.
> 3 - does opensolaris support that graphics chipset (XGI Volari Z9S) well
enough to let me install and get it bootstrapped into operation, after which
I''ll make it headless.
It''ll definately work in a VGA mode. ?Thats all you''ll need
for setup.
> 4 - Are there any gotchas which would keep me from enabling and running the
ECC memory functions productively; I think this is a "no", but as long
as I''m asking questions...
None. ?It''s usually a BIOS config option and transparent to the OS.
> Things of some considerations but lesser importance:
> 5. Electrical power; the E3110 is a low-ish power chip (nominally 65W) Low
power is nice, but not a killer. I''d prefer it to be low, and am
willing to take slow to get more of that, because my file server needs are not
in any way real time. I just need a large, but reliable, bit bucket.
The requirement for ECC limits you from most of the popular
motherboard and processor choices. ?This is one of the few remaining
dividing lines used to separate the marketplace into consumer or
enterprise categories. ?And to enable use of a dual pricing schedule.
> 6. Cost; I''ve been through several iterations of something with an
AMD Athlon 11 X2 240e with an Asus motherboard to get lower power and cost for
the same objectives. I can''t tell that the silly thing would or would
not be something I could make run. And frankly, the data is worth more than the
extra $200 or so for the intel solution - iff the intel solution works. But I
really am not interested in buying a canned commercial solution for a couple of
$K. I''m willing to put in the work setting up and managing the system
in lieu of that, so there is a dollar threshold, I guess.
Personally I get great satisfaction from rolling my own.
>
> And the last silly question. It seems to me that you''d have many,
many adopters if there was a real answer to what the HCL tries to be and
isn''t - an answer to "if I buy this stuff, do I have a prayer of
making it work, or is there a subtle gotcha that''s going to waste my
time and money?" We used to solve that with reference designs. They
don''t have to be perfect, they don''t have to be optimal, but
they should be practical and they should be modestly predictable given moderate
skill in the art.
Agreed - the HCL has not proved to be as useful, in practice, as most
users would like. ?It''s a difficult task - but the typical OpenSolaris
builder is unwilling to put the effort in to contribute to the HCL.
Current OpenSolaris releases work well enough and support enough
current hardware that the risk of something mainstream *not* working
is pretty slim. ?Of course its very inconvenient to install an OS and
then find that it did''nt come with a driver for the onboard NIC,
without the NIC, you can''t get and install the required driver
conveniently.
> I think that an intel and an AMD reference design would be a screaming good
idea for improving the acceptance and population of opensolaris.
Agreed.

Overall I think you''ve done your homework very well. ?I''m sure
others
will contribute alternatives. ?In this post I''ve focused on the topic
of the choices you''ve made, rather than discuss alternatives.

Please keep the list posted on your progress and final config and what
you learn through building and using it.

Regards,

--
Al Hopper ?Logical Approach Inc,Plano,TX al at logical-approach.com
? ? ? ? ? ? ? ? ? Voice: 972.379.2133 Timezone: US CDT
OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/



-- 
Al Hopper  Logical Approach Inc,Plano,TX al at logical-approach.com
                   Voice: 972.379.2133 Timezone: US CDT
OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/

I would suggest a CPU with small L2 cache, as L2 cache will not help a file
server. This allows you use AMD''s new 45W cpu. And 64 bit. 2-4 cores.

And use raidz2.
-- 
This message posted from opensolaris.org

Thanks for replying! I did look into that. The AMD design was my second choice. 

It was :
AMD Athlon II X2 240e (to get low power; the dual core and lack of L3 help
there)
ASUS motherboard (see considerations below)
Cheap VGA? LAN card? This is the mire that ultimately bogged down this one.

Given that data integrity drove me to Opensolaris and zfs for a file server
instead of to a simple NAS box which would have been cheaper and easier, I
reasoned as follows:
- yep, raidz2 is going to meet data integrity better than raidz; so I need plug
in capacity for six disks minimum, that gets to raidz2 with the lowest number of
data integrity issues based on disk failure (I think...)
- there are six-SATA and more motherboards which exist, so there is a premium on
using one of these MBs if opensolaris supports the SATA controller on the MB,
instead of finding and buying one or more disk controller cards. The disk card
which seems most supported and cheapest seems to be the Supermicro PCI-x
eight-SATA version for $100. One could argue that two-three cheaper cards would
get you under $100, but that then begs the issue of complexity, number of slots
on the MB, and additional electrical power use, which is a secondary
consideration, but one which is an obvious issue with the more cards stuffed
into the box. So a six-SATA native controller is a Good Thing
- back at data integrity, a motherboard failure is a problem too; a well-trusted
and well tested MB is a plus. I took that to mean "don''t buy a MB
with "overclocking" mentioned as a plus for it, and don''t
bother with all the fancy onboard widgies you can get. My personal positive
experiences have been with ASUS, Gigabyte, and Intel. I have had motheboard
deaths and erratic behavior with ECS, FCI, and DFI. Haven''t used a
Supermicro, but they seem to be highly recommended.
- gotta have ECC RAM on the MB
- amount of memory is pretty much a don''t care these days; 4gb to 8gb
are probably fine, maybe even less is OK.
- number of slots is only an issue if I have to use external disk cards. This
might be an issue if I was trying to get over a few TB of server storage, but
I''m pretty happy with under 10TB. If I was trying to fill up a 20-30
disk array, I''d have different answers. I can''t afford that
many disks. So a 6-SATA MB is a good compromise.
- intel vs AMD CPUs is a don''t-care to a first approximation. Both
support 64bit, both have ECC support in some flavors. This means select based on
MB features, not architecture. At a secondary level, AMD seems to be cheaper and
perhaps lower power if you get especially the newly announced "e"
versions, notably the Athlon II X2 240e. But saving $100 on a processor or 20W
on the power budget isn''t worth not having the right number of disks on
the MB or hot having chipset drivers be available. I''ve written device
drivers, years ago. I used to wrestle in college, too, but I wouldn''t
want to walk back out on a mat at this point in my life either... 8-)

On the AMD side of things, I''d have gone with the X2 240e I mentioned.
It''s a 45nm chip, AM2+/AM3 socket, 45W TDP. Probably $60-70 if I could
find one in stock; it''s new enough that it''s hard to find. The
competing intel chip is the dual processor Xeon E3110 at 65W, for $180. So AMD
saves me $100 right off the bat.

But what motherboard? All the AMD chips have ECC support inside their chipsets,
so all we have to do is find a MB which lets that work. There are MBs around
which note that they''ll take ECC or non-ECC memory. The question is
whether they *do* anything when there''s an ECC fault. I went through
the entire Gigabyte line and could find no BIOS support for ECC
reporting/action. They may let you use the memory, but they don''t
actually do anything if there''s a fault. This is very much like taking
the spare tire off your car, but feeling OK because there''s a place
where one could go, to me at least.

That gets me down to Intel, ASUS, and Supermicro boards for AMD. ...oops, there
are remarkably few intel-brand MBs which support AMD chips! OK, ASUS and
Supermicro. In the ASUS line, AMD suggests only the ASUS M4A78T-E for the Athlon
II X2. This has only 5 SATA (bad!) but two PCIex X8 slots (good!). The sixth
SATA is routed to an ESATA port on the back of the motherboard. Clumsy, but
could be usable, I guess. The chipset is the AMD 790GX and SB750, with Atheros
(?)L1E LAN and Radeon HD3300 graphics. Downloading and checking the
user''s manual shows that ECC can indeed be set to ... something...
which might work.

And here''s where I diverged from this stack. I had dim results trying
to find confirmation that the chipsets were supported under opensolaris. I got
some indication they did, some that they didn''t, or that they had
problems. Apparently one could insert a VGA card and a LAN card and get around
that. I ... think ... the SATA controller works under opensolaris, and you could
run an ESATA cable back through a hole in the case to get a sixth SATA disk.

This uncertainty is what pushed me back to the intel Xeon stack. 

I''ve thrashed this pretty hard for several weeks now.
-- 
This message posted from opensolaris.org

On Sun, Nov 22, 2009 at 12:43 PM, R.G. Keen <keen at geofex.com> wrote:
> Thanks for replying! I did look into that. The AMD design was my second
> choice.
>
> It was :
> AMD Athlon II X2 240e (to get low power; the dual core and lack of L3 help
> there)
> ASUS motherboard (see considerations below)
> Cheap VGA? LAN card? This is the mire that ultimately bogged down this one.
>
> Given that data integrity drove me to Opensolaris and zfs for a file server
> instead of to a simple NAS box which would have been cheaper and easier, I
> reasoned as follows:
> - yep, raidz2 is going to meet data integrity better than raidz; so I need
> plug in capacity for six disks minimum, that gets to raidz2 with the lowest
> number of data integrity issues based on disk failure (I think...)
> - there are six-SATA and more motherboards which exist, so there is a
> premium on using one of these MBs if opensolaris supports the SATA
> controller on the MB, instead of finding and buying one or more disk
> controller cards. The disk card which seems most supported and cheapest
> seems to be the Supermicro PCI-x eight-SATA version for $100. One could
> argue that two-three cheaper cards would get you under $100, but that then
> begs the issue of complexity, number of slots on the MB, and additional
> electrical power use, which is a secondary consideration, but one which is
> an obvious issue with the more cards stuffed into the box. So a six-SATA
> native controller is a Good Thing
> - back at data integrity, a motherboard failure is a problem too; a
> well-trusted and well tested MB is a plus. I took that to mean
"don''t buy a
> MB with "overclocking" mentioned as a plus for it, and
don''t bother with all
> the fancy onboard widgies you can get. My personal positive experiences
have
> been with ASUS, Gigabyte, and Intel. I have had motheboard deaths and
> erratic behavior with ECS, FCI, and DFI. Haven''t used a
Supermicro, but they
> seem to be highly recommended.
> - gotta have ECC RAM on the MB
> - amount of memory is pretty much a don''t care these days; 4gb to
8gb are
> probably fine, maybe even less is OK.
> - number of slots is only an issue if I have to use external disk cards.
> This might be an issue if I was trying to get over a few TB of server
> storage, but I''m pretty happy with under 10TB. If I was trying to
fill up a
> 20-30 disk array, I''d have different answers. I can''t
afford that many
> disks. So a 6-SATA MB is a good compromise.
> - intel vs AMD CPUs is a don''t-care to a first approximation. Both
support
> 64bit, both have ECC support in some flavors. This means select based on MB
> features, not architecture. At a secondary level, AMD seems to be cheaper
> and perhaps lower power if you get especially the newly announced
"e"
> versions, notably the Athlon II X2 240e. But saving $100 on a processor or
> 20W on the power budget isn''t worth not having the right number of
disks on
> the MB or hot having chipset drivers be available. I''ve written
device
> drivers, years ago. I used to wrestle in college, too, but I
wouldn''t want
> to walk back out on a mat at this point in my life either... 8-)
>
> On the AMD side of things, I''d have gone with the X2 240e I
mentioned. It''s
> a 45nm chip, AM2+/AM3 socket, 45W TDP. Probably $60-70 if I could find one
> in stock; it''s new enough that it''s hard to find. The
competing intel chip
> is the dual processor Xeon E3110 at 65W, for $180. So AMD saves me $100
> right off the bat.
>
> But what motherboard? All the AMD chips have ECC support inside their
> chipsets, so all we have to do is find a MB which lets that work. There are
> MBs around which note that they''ll take ECC or non-ECC memory. The
question
> is whether they *do* anything when there''s an ECC fault. I went
through the
> entire Gigabyte line and could find no BIOS support for ECC
> reporting/action. They may let you use the memory, but they don''t
actually
> do anything if there''s a fault. This is very much like taking the
spare tire
> off your car, but feeling OK because there''s a place where one
could go, to
> me at least.
>
> That gets me down to Intel, ASUS, and Supermicro boards for AMD. ...oops,
> there are remarkably few intel-brand MBs which support AMD chips! OK, ASUS
> and Supermicro. In the ASUS line, AMD suggests only the ASUS M4A78T-E for
> the Athlon II X2. This has only 5 SATA (bad!) but two PCIex X8 slots
> (good!). The sixth SATA is routed to an ESATA port on the back of the
> motherboard. Clumsy, but could be usable, I guess. The chipset is the AMD
> 790GX and SB750, with Atheros (?)L1E LAN and Radeon HD3300 graphics.
> Downloading and checking the user''s manual shows that ECC can
indeed be set
> to ... something... which might work.
>
> And here''s where I diverged from this stack. I had dim results
trying to
> find confirmation that the chipsets were supported under opensolaris. I got
> some indication they did, some that they didn''t, or that they had
problems.
> Apparently one could insert a VGA card and a LAN card and get around that.
I
> ... think ... the SATA controller works under opensolaris, and you could
run
> an ESATA cable back through a hole in the case to get a sixth SATA disk.
>
> This uncertainty is what pushed me back to the intel Xeon stack.
>
> I''ve thrashed this pretty hard for several weeks now.
>
>
Someone can correct me if I''m wrong... but I believe that opensolaris
can do
the ECC scrubbing in software even of the motherboard BIOS doesn''t
support
it.


-- 
--Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091122/2de8bc80/attachment.html>

> Someone can correct me if I'm wrong... but I believe
> that opensolaris can do the ECC scrubbing in software
> even of the motherboard BIOS doesn''t support
> it.
That''s interesting - I didn''t run into that in the background
search.

I suspect that some motherboards just accept the ECC memory bits
from RAM and quietly ignore them, no communication with anything
the CPU can read if an ECC error does happen. I know of several 
organizations I''ve worked with in the past that would just quietly
leave
out the ECC-interrupt line in the interests of "simplification for end 
users" as the resident MBAs would have said. 

My background is hardware, and offhand I can''t think of any way 
that software could fix ECC errors from system RAM if it didn''t 
get notified somehow that one had happened. That doesn''t mean
there''s no way it could happen. 

I''m all over the software repair of bad disk data, though.
That''s one
of the prime motivators for zfs for me. In that case the checksums
are stored in disk where the cpu can get at them. 

Can you point me to a reference?? Anyone? That would open up some 
less expensive hardware if true.
-- 
This message posted from opensolaris.org

Thank you Al! That''s exactly the kind of information
I needed. I very much appreciate the help. 

> It would be helpful to give us a broad description of
> what type of
> data you''re planning on storing. ?Small files, large
> files, required
> capactity etc. ?and we can probably make some
> specific
> recommendations.
I''m storing small files, large-ish files, few GB max. 
Capacity is dollar driven. I can probably continuously 
afford about a half dozen to a dozen hard drives at 
the average price of $100 a drive; that capacity will
work fine for my needs. I''m doing general backup 
and data-save for my house network, about half a 
dozen machines on the house net, each with 1 or 2
hard drives partially used. No editing movies or 
animation, not storing my 1000 ripped DVDs.
> Personally I get great satisfaction from rolling my
> own.
I do too, to the point that my long suffering Significant
Other will avoid mentioning *anything* which I might
want to do myself for fear that it will go on the bottom
of the to-do list. 8-) It''s a good thing I don''t know how
to refine iron ore. Oh, wait... I do! 8-)
> Agreed - the HCL has not proved to be as useful, in
> practice, as most
> users would like. ?It''s a difficult task - but the
> typical OpenSolaris
> builder is unwilling to put the effort in to
> contribute to the HCL.
I''m calmer now... 8-) and I do understand that volunteer 
efforts either get done by zealots or tend to starve. Not
sure which of those is better, but it''s probably accurate. 
> Overall I think you''ve done your homework very well.
> ?I''m sure others
> will contribute alternatives. ?In this post I''ve
> focused on the topic
> of the choices you''ve made, rather than discuss
> alternatives.
Which I appreciate very much. Given that, I may be
able to proceed. I don''t care that I have the optimum
solution, just a workable one that''s affordable and 
current. There are some disciplines where I can 
tell if some system will work by just a casual inspection.
Unfortunately, opensolaris isn''t one of them. 
> Please keep the list posted on your progress and
> final config and what
> you learn through building and using it.
I''ll do that. Again, thanks for the on-target info.
-- 
This message posted from opensolaris.org

On Sun, Nov 22, 2009 at 12:49 PM, Tim Cook <tim at cook.ms>
wrote:
>
.... snip ....
> Someone can correct me if I''m wrong... but I believe that
opensolaris can do
> the ECC scrubbing in software even of the motherboard BIOS doesn''t
support
> it.
The OS is not involved with the ECC functionality of the hardware
AFAIK.   Usually an ECC error will generate a non-maskable interrupt
and the OS will have an appropriate handler setup to handle this
interrupt.

This is the extent of my understanding - without doing additional homework.

Regards,

-- 
Al Hopper  Logical Approach Inc,Plano,TX al at logical-approach.com
                   Voice: 972.379.2133 Timezone: US CDT
OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/

On Sat, November 21, 2009 20:25, Al Hopper wrote:
>> And the last silly question. It seems to me that you''d have
many, many
>> adopters if there was a real answer to what the HCL tries to be and
>> isn''t - an answer to "if I buy this stuff, do I have a
prayer of making
>> it work, or is there a subtle gotcha that''s going to waste my
time and
>> money?" We used to solve that with reference designs. They
don''t have to
>> be perfect, they don''t have to be optimal, but they should be
practical
>> and they should be modestly predictable given moderate skill in the
art.
>
> Agreed - the HCL has not proved to be as useful, in practice, as most
> users would like. ?It''s a difficult task - but the typical
OpenSolaris
> builder is unwilling to put the effort in to contribute to the HCL.
> Current OpenSolaris releases work well enough and support enough
> current hardware that the risk of something mainstream *not* working
> is pretty slim. ?Of course its very inconvenient to install an OS and
> then find that it did''nt come with a driver for the onboard NIC,
> without the NIC, you can''t get and install the required driver
> conveniently.
The problem of course is how quickly the hardware world is still moving.

Is there enough information available from system configuration utilities
to make an automatic HCL (or unofficial HCL competitor) feasible?  Someone
could write an application people could run which would report their
opinion on how well it works, plus the self-reported identity of all key
components?  (It could report uptime, too, as one very small objective
rating of stability.)
-- 
David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

On 11/23/09 10:10 AM, David Dyer-Bennet wrote:
> Is there enough information available from system configuration utilities
> to make an automatic HCL (or unofficial HCL competitor) feasible?  Someone
> could write an application people could run which would report their
> opinion on how well it works, plus the self-reported identity of all key
> components?  (It could report uptime, too, as one very small objective
> rating of stability.)
IIRC, the HCL doesn''t really talk about applications. We have some
really
flaky PCs that run Open Solaris beautifully and their uptime is measured
in months (basically only new releases or long power cuts make them
come down). Would I recommend them for a ZFS based server? Not a
chance! But they make super reliable X-Terminals...

As Richard Elling has pointed out so eloquently, a reliable storage
system has to be engineered to minimize or eliminate SPoFS, and I
doubt you''ll ever find that on an HCL, which really serves a different
purpose, IMO.

Cheers -- Frank

On Mon, November 23, 2009 09:53, Frank Middleton wrote:
> On 11/23/09 10:10 AM, David Dyer-Bennet wrote:
>
>> Is there enough information available from system configuration
>> utilities
>> to make an automatic HCL (or unofficial HCL competitor) feasible?
>> Someone
>> could write an application people could run which would report their
>> opinion on how well it works, plus the self-reported identity of all
key
>> components?  (It could report uptime, too, as one very small objective
>> rating of stability.)
>
> IIRC, the HCL doesn''t really talk about applications. We have some
really
> flaky PCs that run Open Solaris beautifully and their uptime is measured
> in months (basically only new releases or long power cuts make them
> come down). Would I recommend them for a ZFS based server? Not a
> chance! But they make super reliable X-Terminals...
So they''re okay except maybe for IO?  Well, that''s the sort of
thing
people could add comments on as they got experience with them.  Still
seems like it could be useful.
> As Richard Elling has pointed out so eloquently, a reliable storage
> system has to be engineered to minimize or eliminate SPoFS, and I
> doubt you''ll ever find that on an HCL, which really serves a
different
> purpose, IMO.
Lots of storage servers, outside the big corporate environment, can''t
afford full-blown redundancy.  For many of us, we''re just taking the
first
steps into using any kind of redundancy at all in disks for our file
servers.  Full enterprise-grade storage is too expensive for many of us,
and we''re looking for close to that level risk of loss, but are willing
to
sacrifice some on the availability (it''s okay for the server to be down
while I replace the power supply; after all, I can''t be sitting at my
computer anyway, I have to replace this power supply).
-- 
David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

Your point is well taken, Frank, and I agree - there has to be some serious
design work for reliability. My background includes both hardware design for
reliability and field service engineering support, so the issues are not at all
foreign to me. Nor are the limits of something like a volunteer reporting
mechanism like the HCL.

One reason for the existence of this forum is the advocacy of opensolaris and
presumably its expansion into the computing world. While many of the people who
post here are undoubtedly working professionals in the solaris/opensolaris and
*nix worlds, the number is limited. But there is a much larger number of people
who are fully capable of building and setting up opensolaris systems who would
love to do so given the promise of zfs as an application. They don''t
(yet) love the OS, they want the tool for other things. They might well come to
love the OS if they got the tool.

The mystery (as viewed by a beginner/outsider) involved in getting a small,
affordable, usable and modestly reliable data storage system set up is
preventing the inclusion of a whole lot of prospective converts. Nexenta has
noted this, and is presumably making money solving this issue for people. Look
what happened to the PC world when DIY hardware/OS setups happened.

So I think that making reference designs available in some way is a Good Thing
for the opensolaris community to prosper.

But what do I know?  8-)
-- 
This message posted from opensolaris.org

> > On 11/23/09 10:10 AM, David Dyer-Bennet wrote:
> Lots of storage servers, outside the big corporate 
>environment, can''t
> afford full-blown redundancy.  For many of us, we''re
> just taking the first
> steps into using any kind of redundancy at all in
> disks for our file
> servers.  Full enterprise-grade storage is too
> expensive for many of us,
> and we''re looking for close to that level risk of
> loss, but are willing to
> sacrifice some on the availability (it''s okay for the
> server to be down
> while I replace the power supply; after all, I can''t
> be sitting at my
> computer anyway, I have to replace this power
> supply).
What you said! And have a beer!
-- 
This message posted from opensolaris.org

>>>>> "tc" == Tim Cook <tim at cook.ms> writes:
    tc> I believe that opensolaris can do the ECC scrubbing in
    tc> software even of the motherboard BIOS doesn''t support it.

yeah, I don''t really understand how the solaris idle page scrubbing
interacts with whatever.  scrubbing''s a hardware feature for AMD.  It
needs to be turned on, but you can do this before or after boot:

 http://hyvatti.iki.fi/~jaakko/sw/

There''s scrubbing for both DRAM and cache.

Solaris does need to report the errors somehow, and I''ve not tested
that.  This is a problem, but my WAG is that it works or doesn''t for
AMD, irrespective of the BIOS.  The very old script above worked with
phenom II 910 (new, 45nm) therefore it seems AMD has not changed the
knobs much.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 304 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091123/e40f785d/attachment.bin>

On Nov 23, 2009, at 12:48 PM, Miles Nordin wrote:
>>>>>> "tc" == Tim Cook <tim at cook.ms>
writes:
>
>    tc> I believe that opensolaris can do the ECC scrubbing in
>    tc> software even of the motherboard BIOS doesn''t support
it.
>
> yeah, I don''t really understand how the solaris idle page
scrubbing
> interacts with whatever.
It has been a while since I looked, but it can be as simple as
just reading everything in a loop.  The trick is that some processors
(e.g. modern SPARC) can do non-cache-polluting reads for main
memory, which is what you want to do for a scrub.  Each proc can
be a little different, but the general algorithm is described in
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/sun4u/os/memscrub.c

Page retire is intimately related, and has a decent description in
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/vm/page_retire.c
  -- richard

Most ECC setups are as you describe. The memory hardware detects and corrects
all 1-bit errors, and detects all two-bit errors on its own. What ... should ...
happen is that the OS should get an interrupt when this happens so it has the
opportunity to note the error in logs and to higher level stuff if needed - map
out the memory in question, call an operator, halt and catch fire, etc. But the
hardware must have that interrupt line connected to something to even make this
possible. And the OS doesn''t have to do anything, necessarily.

Although, as mentioned, if the OS has a low level read-through-memory routine,
it does guarantee that memory is scrubbed of one-bit error and bad pages found.
-- 
This message posted from opensolaris.org

Most ECC setups are as you describe. The memory hardware detects and corrects
all 1-bit errors, and detects all two-bit errors on its own. What ... should ...
happen is that the OS should get an interrupt when this happens so it has the
opportunity to note the error in logs and to higher level stuff if needed - map
out the memory in question, call an operator, halt and catch fire, etc. But the
hardware must have that interrupt line connected to something to even make this
possible. And the OS doesn''t have to do anything, necessarily.

Although, as mentioned, if the OS has a low level read-through-memory routine,
it does guarantee that memory is scrubbed of one-bit error and bad pages found.
-- 
This message posted from opensolaris.org