zfs discuss - Sep 2010 - non-ECC Systems and ZFS for home users (was: Please warn a home user against OpenSolaris under VirtualBox under WinXP ; ))

Hi.

2010/9/19 R.G. Keen <keen at geofex.com>
> and last-generation hardware is very, very cheap.
Yes, of course, it is. But, actually, is that a true statement? I''ve
read
that it''s *NOT* advisable to run ZFS on systems which do NOT have ECC
RAM. And those cheapo last-gen hardware boxes quite often don''t have
ECC, do they?

So, I wonder - what''s the recommendation, or rather, experience as far
as home users are concerned? Is it "safe enough" now do use ZFS on
non-ECC-RAM systems (if backups are around)?

Regards,

Alexander
--
???? Lifestream (Twitter, Blog, ?) ??http://alexs77.soup.io/? ?? ?
? Chat (Jabber/Google Talk) ? a.skwar at gmail.com , AIM: alexws77? ?

On 09/23/10 06:33 PM, Alexander Skwar wrote:
> Hi.
>
> 2010/9/19 R.G. Keen<keen at geofex.com>
>
>    
>> and last-generation hardware is very, very cheap.
>>      
> Yes, of course, it is. But, actually, is that a true statement?
I''ve read
> that it''s *NOT* advisable to run ZFS on systems which do NOT have
ECC
> RAM. And those cheapo last-gen hardware boxes quite often don''t
have
> ECC, do they?
>
> So, I wonder - what''s the recommendation, or rather, experience as
far
> as home users are concerned? Is it "safe enough" now do use ZFS
on
> non-ECC-RAM systems (if backups are around)?
>
>    
It''s as safe as running any other OS.

The big difference is ZFS will tell you when there''s a corruption. 
Most
users of other systems are blissfully unaware of data corruption!

All my desktops use ZFS, none have ECC.

-- 
Ian.

I''m using ZFS on a system w/o ECC; it works (it''s an Atom
230).

Note that this is not different from using another OS; the difference is 
that ZFS will complain when memory leads to disk corruption; without ZFS 
you will still have memory corruption but you wouldn''t know.

Is it helpful not knowing that you have memory corruption?  I don''t
think
so.

I''ve love to have a small (<40W) system with ECC but it is difficult
to
find one.

Casper

On 23-9-2010 10:25, Casper.Dik at Sun.COM wrote:
> I''m using ZFS on a system w/o ECC; it works (it''s an Atom
230).
I''m using ZFS on a non-ECC machine for years now without any issues. 
Never had errors. Plus, like others said, other OS''ses have the same 
problems and also run quite well. If not, you don''t know it. With ZFS 
you will know.
I would say - just go for it. You will never want to go back.

>  On 23-9-2010 10:25, Casper.Dik at Sun.COM wrote:
>> I''m using ZFS on a system w/o ECC; it works (it''s an
Atom 230).
>
>I''m using ZFS on a non-ECC machine for years now without any
issues.
>Never had errors. Plus, like others said, other OS''ses have the
same
>problems and also run quite well. If not, you don''t know it. With
ZFS
>you will know.
>I would say - just go for it. You will never want to go back.

Indeed.  While I mirror stuff on the same system, I''m now also making
backups using a USB connected disk (eSATA would be better but the box
only has USB).

My backup consists of:



for pool in $pools
do

	zfs snapshot -r $pool@$newnapshot
	zfs send -R -I $pool@$lastsnapshot $pool@$newsnapshot |
		zfs receive -v -u -d -F portable/$pool
done

then I export and store the portable pool somewhere else.

I do run a once per two weeks scrub for all the pools, just in case.

Casper

On 09/23/10 03:01, Ian Collins wrote:
>> So, I wonder - what''s the recommendation, or rather,
experience as far
>> as home users are concerned? Is it "safe enough" now do use
ZFS on
>> non-ECC-RAM systems (if backups are around)?
>>
> It''s as safe as running any other OS.
>
> The big difference is ZFS will tell you when there''s a corruption.
Most
> users of other systems are blissfully unaware of data corruption!
This runs you into the possibility of perfectly good files becoming inaccessible
due to bad checksums being written to all the mirrors. As Richard Elling
wrote some time ago in "[zfs-discuss] You really do need ECC RAM", see
http://www.cs.toronto.edu/%7Ebianca/papers/sigmetrics09.pdf. There
were a couple of zfs-discuss threads quite recently about memory problems
causing serious issues. Personally, I wouldn''t trust any valuable data
to any
system without ECC, regardless of OS and file systems. For home use, used
Suns are available at ridiculously low prices and they seem to be much better
engineered than your typical PC. Memory failures are much more likely than
winning the pick 6 lotto...

FWIW Richard helped me diagnose a problem with checksum failures on
mirrored drives a while back and it turned out to be the CPU itself getting
the actual checksum wrong /only on one particular file/, and even then only
when the ambient temperature was high. So ZFS is good at ferreting out
obscure hardware problems :-).

Cheers -- Frank

On 23-9-2010 16:34, Frank Middleton wrote:
>  For home use, used Suns are available at ridiculously low prices and
>  they seem to be much better engineered than your typical PC. Memory
>  failures are much more likely than winning the pick 6 lotto...
And about what SUN systems are you thinking for ''home use'' ?
The likeliness of memory failures might be much higher than becoming a 
millionair, but in the years past I have never had one. And my home 
sytems are rather cheap. Mind you, not the cheapest, but rather cheap. I 
do buy good memory though. So, to me, with a good backup I feel rather 
safe using ZFS. I also had it running for quite some time on a 32bits 
machine and that also worked out fine.

The fact that a perfectly good file can not be read because of a bad 
checksum is a design failure imho. There should be an option to overrule 
this behaviour of ZFS.

My 2?t

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100923/dc13b4eb/attachment.html>

On Sep 23, 2010, at 9:08 AM, Dick Hoogendijk wrote:
> On 23-9-2010 16:34, Frank Middleton wrote:
> 
> > For home use, used Suns are
>       available at ridiculously low prices and
> 
>       > they seem to be much better engineered than your typical PC.
>       Memory
> 
>       > failures are much more likely than winning the pick 6
>       lotto...
> 
> And about what SUN systems are you thinking for ''home
use'' ?
At one time, due to market pricing pressure, Sun actually sold a server without
ECC.
Bad idea, didn''t last long.  Unfortunately, the PeeCee market is just
too cheap to
value ECC.  So they take the risk and hope for the best.
> The likeliness of memory failures might be much higher than becoming a
millionair, but in the years past I have never had one. And my home sytems are
rather cheap. Mind you, not the cheapest, but rather cheap. I do buy good memory
though. So, to me, with a good backup I feel rather safe using ZFS. I also had
it running for quite some time on a 32bits machine and that also worked out
fine.
Part of the difference is the expected use.  For PCs which are only used 8 hours
per
day, 40 hours per week, rebooting regularly, the risk of transient main memory
errors
is low.  For servers running 24x7, rebooting once a year, the risk is much
higher.
> The fact that a perfectly good file can not be read because of a bad
checksum is a design failure imho. There should be an option to overrule this
behaviour of ZFS.
It isn''t a perfectly good file once it has been corrupted. But there
are some ways to get
at the file contents.  Remember, the blocks are checksummed, not the file. So if
a bad
block is in the file, you can skip over it.
http://blogs.sun.com/relling/entry/holy_smokes_a_holey_file
http://blogs.sun.com/relling/entry/dd_tricks_for_holey_files
http://blogs.sun.com/relling/entry/more_on_holey_files

 -- richard

-- 
OpenStorage Summit, October 25-27, Palo Alto, CA
http://nexenta-summit2010.eventbrite.com
ZFS and performance consulting
http://www.RichardElling.com

On Thu, September 23, 2010 01:33, Alexander Skwar wrote:
> Hi.
>
> 2010/9/19 R.G. Keen <keen at geofex.com>
>
>> and last-generation hardware is very, very cheap.
>
> Yes, of course, it is. But, actually, is that a true statement?
I''ve read
> that it''s *NOT* advisable to run ZFS on systems which do NOT have
ECC
> RAM. And those cheapo last-gen hardware boxes quite often don''t
have
> ECC, do they?
Last-generation server hardware supports ECC, and was usually populated
with ECC.  Last-generation desktop hardware rarely supports ECC, and was
even more rarely populated with ECC.

The thing is, last-generation server hardware is, um, marvelously adequate
for most home setups (the problem *I* see with it, for many home setups,
is that it''s *noisy*).  So, if you can get it cheap in a sound-level
that
fits your needs, that''s not at all a bad choice.

I''m running a box I bought new as a home server, but it''s NOW
at least
last-generation hardware (2006), and it''s still running fine; in
particular the CPU load remains trivial compared to what the box supports
(not doing compression or dedup on the main data pool, though I do
compress the backup pools on external USB disks).  (It does have ECC; even
before some of the cases leading to that recommendation were explained on
that list, I just didn''t see the percentage in not protecting the
memory.)

-- 
David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

I should clarify. I was addressing just the issue of 
virtualizing, not what the complete set of things to
do to prevent data loss is. 
> 2010/9/19 R.G. Keen <keen at geofex.com>
> > and last-generation hardware is very, very cheap.
> Yes, of course, it is. But, actually, is that a true
> statement? 
Yes, it is. Last-generation hardware is, in general, 
very cheap. But there is no implication either way 
about ECC in that. And in fact, there is a buyer''s
market for last-generation *servers* with ECC that 
is very cheap too. I can get a single-unit rackmount
server setup for under $100 here in Austin that includes
ECC memory. 

That may not be the best of all possible things to do
on a number of levels. But for me, the likelihood of 
making a setup or operating mistake in a virtual machine 
setup server is far outweighs the hardware cost to put
another physical machine on the ground. 
>I''ve read that it''s *NOT* advisable to run ZFS on systems 
>which do NOT have ECC  RAM. And those cheapo last-gen 
>hardware boxes quite often don''t have ECC, do they?
Most of them, the ex-desktop boxes, do not. However, 
as I noted above, removed-from-service servers are also
quite cheap. They *do* have ECC. I say this just to 
illustrate the point that a statement about last generation
hardware says nothing about ECC, either positive or negative.

In fact, the issue goes further. Processor chipsets from both
Intel and AMD used to support ECC on an ad-hoc basis. It may
have been there, but may or may not have been supported
by the motherboard. Intels recent chipsets emphatically do 
not support ECC. AMDs do, in general. However, the motherboard
must still support the ECC reporting in hardware and BIOS for
ECC to actually work, and you have to buy the ECC memory. 
The newer the intel motherboard, the less likely and more
expensive ECC is. Older intel motherboards sometimes
did support ECC, as a side note. 

There''s about sixteen more pages of typing to cover the issue 
even modestly correctly. The bottom line is this: for 
current-generation hardware, buy an AMD AM3 socket CPU,
ASUS motherboard, and ECC memory. DDR2 and DDR3 ECC
memory is only moderately more expensive than non-ECC.

I have this year built two Opensolaris servers from scratch.
They use the Athlon II processors, 4GB of ECC memory and
ASUS motherboards. This setup runs ECC, and supports ECC 
reporting and scrubbing. The cost of this is about $65 for
the CPU, $110 for memory, and $70-$120 for the motherboard. 
$300 more or less gets you new hardware that runs a 64bit
OS, ECC, and zfs, and does not give you worries about the 
hardware going into wearout. I also bought new, high quality
power supplies for $40-$60 per machine because the power
supply is a single point of failure, and wears out - that''s a 
fact that many people ignore until the machine doesn''t come
up one day.
> So, I wonder - what''s the recommendation, or rather,
> experience as far as home users are concerned? Is it "safe 
>enough" now do use ZFS on non-ECC-RAM systems (if backups 
>are around)?
That''s more a question about how much you trust your backups
than a question about ECC. 

ZFS is a layer of checking and recovery on disk writes. If your
memory/CPU tell it to carefully save and recover corrupted
data, it will. Memory corruption is something zfs does not 
address in any way, positive or negative. 

[i][b]The correct question is this: given how much value you put on
not losing your data to hardware or software errors, how much
time and money are you willing to spend to make sure you don''t 
lose your data?[/b][/i]
ZFS prevents or mitigates many of the issues involved with disk
errors and bit rot. ECC prevents or mitigates many of the issues
involved with memory corruption. 

My recommendation is this: if you are playing around, fine, use
virtual machines for your data backup. If you want some amount
of real data backup security, address the issues of data corruption
on as many levels as you can. "Safe enough" is something only 
you can answer. My answer, for me and my data, is a separate
machine which does only data backup, which runs both ECC and
zfs, on new (and burnt-in) hardware, which runs only the data
management tasks to simplify the software interactions being run,
and that being two levels deep on different hardware setups, 
finally flushing out to offline DVDs which are themselves protected
by ECC (look up DVDisaster) and periodically scanned for errors
and recopied. 

That probably seems excessive. But I''ve been burned with subtle
data loss before. It only takes one or two flipped bits in the wrong
places to make a really ugly scenario. Losing an entire file is in 
many ways easier to live with than a quiet error that gets 
propagated silently into your backup stream. When that happens, 
you can''t trust **any** file until you have manually checked it, if
that is even possible. Want a really paranoia inducing situation?
Think about what happens if you find a silent bit corruption in 
a file system that includes encrypted files. 

So - what''s your data worth to you?
-- 
This message posted from opensolaris.org

[I''m deleting the whole thread, since this is a rehash of several 
discussions on this list previously - check out the archives, and search 
for "ECC RAM"]


These days, for a "home" server, you really have only one choice to
make:

"How much power do I care that this thing uses?"



If you are sensitive to the power (and possibly cooling) budget that 
your home server might use, then there are a myriad of compromises 
you''re going to have to make - and lack of ECC support is almost 
certainly going to be the first one.  Very, very, very few low-power 
(i.e. under 25W) CPUs support ECC.  A couple of the very-low-voltage EE 
Opterons, and some of the laptop-series Core2 chips are about the best 
hope you have get a CPU which is both low-power and supports ECC.


If you don''t really care about ultra-low-power, then there''s
absolutely
no excuse not to buy a USED server-class machine which is 1- or 2- 
generations back.  They''re dirt cheap, readily available, and support 
all those nice features you''ll have problems replicating in trying to
do
a build-it-yourself current-gen box.


For instance, an IBM x3500 tower machine, with dual-core Xeon 
5100-series CPUs, on-board ILOM/BMC, redundant power supply, ECC RAM 
support, and 8 hot-swap SAS/SATA 3.5" bays (and the nice SAS/SATA 
controller supported by Solaris) is about $500, minus the drives.  The 
Sun Ultra 40 is similar.  The ultra-cheapo Dell 400SC works fine, too.


And, frankly, buying a used brand-name server machine will almost 
certainly give you a big advantage over building-it-yourself in one 
crucial (and generally overlooked) area:  the power supply.   These 
machines have significantly better power supplies (and, most have 
redundant ones) than what you can buy for a PC.  Indeed, figure you need 
to spend at least $100 on the PS alone if you build it yourself.


-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA

On 9/23/2010 at 12:38 PM Erik Trimble wrote:

| [snip]
|If you don''t really care about ultra-low-power, then there''s
absolutely 
|no excuse not to buy a USED server-class machine which is 1- or 2- 
|generations back.  They''re dirt cheap, readily available, 
| [snip]
 ============


Anyone have a link or two to a place where I can buy some dirt-cheap,
readily available last gen servers?

On 2010-Sep-24 00:58:47 +0800, "R.G. Keen" <keen at geofex.com>
wrote:
>That may not be the best of all possible things to do
>on a number of levels. But for me, the likelihood of 
>making a setup or operating mistake in a virtual machine 
>setup server is far outweighs the hardware cost to put
>another physical machine on the ground. 
The downsides are generally that it''ll be slower and less power-
efficient that a current generation server and the I/O interfaces will
be also be last generation (so you are more likely to be stuck with
parallel SCSI and PCI or PCIx rather than SAS/SATA and PCIe).  And
when something fails (fan, PSU, ...), it''s more likely to be customised
in some way that makes it more difficult/expensive to repair/replace.
>In fact, the issue goes further. Processor chipsets from both
>Intel and AMD used to support ECC on an ad-hoc basis. It may
>have been there, but may or may not have been supported
>by the motherboard. Intels recent chipsets emphatically do 
>not support ECC.
Not quite.  When Intel moved the memory controllers from the
northbridge into the CPU, they made a conscious decision to separate
server and desktop CPUs and chipsets.  The desktop CPUs do not support
ECC whereas the server ones do - this way they can continue to charge
a premium for "server-grade" parts and prevent the server
manufacturers from using lower-margin desktop parts.  This means that
if you want an Intel-based solution, you need to look at a Xeon CPU.
That said, the low-end Xeons aren''t outrageously expensive and you
generally wind up with support for registered RAM and registered ECC
RAM is often easier to find than unregistered ECC RAM.
> AMDs do, in general.
AMD chose to leave ECC support in almost all their higher-end memory
controllers, rather than use it as a market differentiator.  AFAIK,
all non-mobile Athlon, Phenom and Opteron CPUs support ECC, whereas
the lower-end Sempron, Neo, Turion and Geode CPUs don''t.  Note that
Athlon and Phenom CPUs normally need unbuffered RAM whereas Opteron
CPUs normally want buffered/registered RAM.
> However, the motherboard
>must still support the ECC reporting in hardware and BIOS for
>ECC to actually work, and you have to buy the ECC memory. 
In the case of AMD motherboards, it''s really just laziness on the
manufacturer''s part to not bother routing the additional tracks.
>The newer the intel motherboard, the less likely and more
>expensive ECC is. Older intel motherboards sometimes
>did support ECC, as a side note. 
On older Intel motherboards, it was a chipset issue rather than a
CPU issue (and even if the chipset supported ECC, the motherboard
manufacturer might have decided to not bother running the ECC tracks).
>There''s about sixteen more pages of typing to cover the issue 
>even modestly correctly. The bottom line is this: for 
>current-generation hardware, buy an AMD AM3 socket CPU,
>ASUS motherboard, and ECC memory. DDR2 and DDR3 ECC
>memory is only moderately more expensive than non-ECC.
Asus appears to have made a conscious decision to support ECC on
all AMD motherboards whereas other vendors support it sporadically
and determining whether a particular motherboard supports ECC can
be quite difficult since it''s never one of the options in their
motherboard selection tools.

And when picking the RAM, make sure it''s compatible with your
motherboard - motherboards are virtually never compatible with
both unbuffered and buffered RAM.
>hardware going into wearout. I also bought new, high quality
>power supplies for $40-$60 per machine because the power
>supply is a single point of failure, and wears out - that''s a 
>fact that many people ignore until the machine doesn''t come
>up one day.
"Doesn''t come up one day" is at least a clear failure.  With
a
cheap (or under-dimensioned) PSU, things are more likely to go
out of tolerance under heavy load so you wind up with unrepeatable
strange glitches.
>Think about what happens if you find a silent bit corruption in 
>a file system that includes encrypted files. 
Or compressed files.

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100924/8e098446/attachment.bin>

> On 2010-Sep-24 00:58:47 +0800, "R.G. Keen"
> <keen at geofex.com> wrote:
> > But for me, the likelihood of
> >making a setup or operating mistake in a virtual machine 
> >setup server is far outweighs the hardware cost to put
> >another physical machine on the ground. 
> 
> The downsides are generally that it''ll be slower and less power-
> efficient that a current generation server 
My comment was about a physical machine versus virtual machine, 
and my likelihood of futzing up the setup, not new machine versus 
old machine. There are many upsides and downsides on the new 
versus old questions too. 
>and the I/O interfaces will
> be also be last generation (so you are more likely to be stuck with
> parallel SCSI and PCI or PCIx rather than SAS/SATA and PCIe).  And
> when something fails (fan, PSU, ...), it''s more likely to be
customised
> in some way that makes it more difficult/expensive to repair/replace.
Presuming what you did was buy a last generation server after you 
decided to go for a physical machine. That''s not what I did, as I
mentioned
later in the posting. Server hardware in general is more expensive than
desktop, and even last generation server hardware will cost more
to repair than desktop. To a hardware manufacturer, "server" is 
synonymous with "these guys can be convinced to pay more if we make
something a little different". And there is a cottage industry of people 
who sell repair parts for older servers at exorbitant prices because there
is some non-techie businessman who will pay.
> Not quite.  When Intel moved the memory controllers from the 
> northbridge into the CPU, they made a conscious  decision to separate
> server and desktop CPUs and chipsets.  The desktop CPUs do not support
> ECC whereas the server ones do 
So, from the lower-cost new hardware view, newer Intel chipsets emphatically
do not support ECC. The (expensive) server-class hardware/chipsets, etc., do. 
A lower-end home class server is unlikely to be built from these much more
expensive - by plan/design - parts. 
> That said, the low-end Xeons aren''t outrageously expensive 
They aren''t. I considered using  Xeons in my servers. It was about
another
$200 in the end. I bought disks with the $200. 
>and you
> generally wind up with support for registered RAM and registered ECC
> RAM is often easier to find than unregistered ECC RAM.
I had no difficulty at all finding unregistered ECC RAM. 
Newegg has steady stock of DDR2 and DDR3 unregistered and registered
ECC. For instance: 
2GB 240-Pin DDR3 ECC Registered KVR1333D3D8R9S/2GHT is $59.99.
2GB 240-Pin DDR3 1333 ECC Unbuffered Server Memory Model KVR1333D3E9S/2G
is $43.99 plus $2.00 shipping. 
2GB 240-pin DDR3 NON-ECC is available for $35 per stick and up. The Kingston
brand I used in the ECC examples is $40.
These are representative, and there are multiple choices, in stock, for all
three
categories. "Intel certified" costs more if you get registered.
> > AMDs do, in general.
> AMD chose to leave ECC support in almost all their higher-end memory
> controllers, rather than use it as a market differentiator. AFAIK,
> all non-mobile Athlon, Phenom and Opteron CPUs support ECC, whereas
> the lower-end Sempron, Neo, Turion and Geode CPUs don''t.  
I guess I should have looked at the lower end CPUs - and chipsets before
I took my "in general" count. I didn''t, and every chipset I
saw had ECC
support. My lowest end CPU was the Athlon II X2 240e, and every chipset
for that and above that I found supports ECC. 
> In the case of AMD motherboards, it''s really just laziness on the
> manufacturer''s part to not bother routing the additional tracks.
And doing the support in bios. I did research these issues a fair amount.
For the same chipset, ASUS MBs seem to have bios settings for ECC and
Gigabyte, for instance, do not. I determined this by downloading the 
user manuals for the mobos and reading them. I didn''t find a brand
other
that ASUS that had a clear support for ECC in the bios. 

But my search was not exhaustive. 
> On older Intel motherboards, it was a chipset issue rather than a
> CPU issue (and even if the chipset supported ECC, the motherboard
> manufacturer might have decided to not bother running
> the ECC tracks).
I think that''s generically true.
> Asus appears to have made a conscious decision to support ECC on
> all AMD motherboards whereas other vendors support it sporadically
> and determining whether a particular motherboard supports ECC can
> be quite difficult since it''s never one of the options in their
> motherboard selection tools.
Yep. I resorted to selecting mobos that I''d otherwise want, then 
downloaded the user manuals and read the bios configurations 
pages. If it didn''t specifically say how to configure ECC, they it 
MIGHT be supported somehow, but also might not. Gigabyte boards
for instance were reputed to support ECC in an undocumented way, 
but I figured if they didn''t want to say how to configure it, they sure
wouldn''t have worried about testing whether it worked. 

> And when picking the RAM, make sure it''s compatible with your
> motherboard - motherboards are virtually never compatible with
> both unbuffered and buffered RAM.
Yep. Once I selected mobos on features other than ECC, I investigated
ECC, and where that was supported, researched the issue of the specific
ECC memory brand on the mobo site and memory vendor site. 

There was a premium price attached to both registered memory sticks
and the mobos that needed them. I guess what amounts to the mid
range chipsets for AMD all used unbuffered/unregistered ECC memory.
The higher end CPUS were out of my selection criteria because of the
heat and power load, so I didn''t find the registered ECC mobos in AMD
chipsets. 
> >Think about what happens if you find a silent bit corruption in 
> >a file system that includes encrypted files. 
> Or compressed files.
That''ll do it too. I personally don''t compress files I think
it''s critical
that I keep. That side steps the issue of things like images which 
normally contain compression, but I had some bad experiences 
which taught me the similarity between corrupted compressed 
files and encrypted files for which you don''t have the password. 
I''m probably too conservative that way.
-- 
This message posted from opensolaris.org

On 09/23/10 19:08, Peter Jeremy wrote:
> The downsides are generally that it''ll be slower and less power-
> efficient that a current generation server and the I/O interfaces will
> be also be last generation (so you are more likely to be stuck with
> parallel SCSI and PCI or PCIx rather than SAS/SATA and PCIe).  And
> when something fails (fan, PSU, ...), it''s more likely to be
customised
> in some way that makes it more difficult/expensive to repair/replace.
Sometimes the bargains on E-Bay are such that you can afford to get
2 or even a 3rd machine for spares, and a PCI-X SAS card has more
than adequate performance for SOHO use. But, I agree, repair is
probably impossible unless you can simply swap in a spare part from
another box. However server class machines are pretty tough. My used
Sun hardware has yet to drop a beat and they''ve been running 24*7
for years - well, I cycle the spares since they were never needed for
parts, so it''s less than that. But they are noisy...

Surely the issue about repairs extends to current generation hardware.
It gets obsolete so quickly that finding certain parts (especially mobos)
may be next to impossible. So what''s the difference other than lots of
$$$?

Cheers -- Frank

On 9/24/2010 6:27 AM, Frank Middleton wrote:
> On 09/23/10 19:08, Peter Jeremy wrote:
>
>> The downsides are generally that it''ll be slower and less
power-
>> efficient that a current generation server and the I/O interfaces will
>> be also be last generation (so you are more likely to be stuck with
>> parallel SCSI and PCI or PCIx rather than SAS/SATA and PCIe).  And
>> when something fails (fan, PSU, ...), it''s more likely to be
customised
>> in some way that makes it more difficult/expensive to repair/replace.
>
> Sometimes the bargains on E-Bay are such that you can afford to get
> 2 or even a 3rd machine for spares, and a PCI-X SAS card has more
> than adequate performance for SOHO use. But, I agree, repair is
> probably impossible unless you can simply swap in a spare part from
> another box. However server class machines are pretty tough. My used
> Sun hardware has yet to drop a beat and they''ve been running 24*7
> for years - well, I cycle the spares since they were never needed for
> parts, so it''s less than that. But they are noisy...
>
> Surely the issue about repairs extends to current generation hardware.
> It gets obsolete so quickly that finding certain parts (especially mobos)
> may be next to impossible. So what''s the difference other than
lots of
> $$$?
>
> Cheers -- Frank
>
>
Most certainly, but remember that even 2- generations old hardware at 
this point means either a 2200-series Opteron or a 5100-series Xeon 
system, which come with a minimum of PCI-E 1.0 slots, DDR2 RAM, and 
usually SAS controllers.   I''ve a dual-socket Barcelona (Opteron 2354) 
system here under my desk, and it''s so overkill for a SOHO server
it''s
not even funny.

Also, if you get OEM (name-brand) equipment from a used seller, that 
means you get the ability to search eBay (or your favorite local 
recycler) for the FRU part that goes bad.  There''s a *ton* of spare 
parts floating around the used market for anything less than 5 years 
old, and even the 5-8 year-old parts are commonplace.   HP, Sun, IBM, 
and Dell all have the FRU/Option part label *on the part itself*, so if 
something dies, it''s idiot simple to figure out what to get to replace 
it.  And, the used parts prices are, well, *nice*.


Honestly, I''ve said it before, and I''ll say it (yet) again: 
unless you
have very stringent power requirement (or some other unusual 
requirement, like very, very low noise),  used (or even new-in-box, 
previous generation excess inventory) OEM stuff is far superior to any 
build-it-yourself rig you can come up with. So much so, in fact, that we 
should really consider the reference recommendation for a ZFS fileserver 
to be certain configs of brand-name hardware, and NOT try to recommend 
other things to folks.


-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA

On 09/25/10 02:54 AM, Erik Trimble wrote:
>
> Honestly, I''ve said it before, and I''ll say it (yet)
again:  unless
> you have very stringent power requirement (or some other unusual 
> requirement, like very, very low noise),  used (or even new-in-box, 
> previous generation excess inventory) OEM stuff is far superior to any 
> build-it-yourself rig you can come up with. So much so, in fact, that 
> we should really consider the reference recommendation for a ZFS 
> fileserver to be certain configs of brand-name hardware, and NOT try 
> to recommend other things to folks.
>
Unless you live somewhere with a very small used server market that is!

-- 
Ian.

On 9/25/2010 1:57 AM, Ian Collins wrote:
> On 09/25/10 02:54 AM, Erik Trimble wrote:
>>
>> Honestly, I''ve said it before, and I''ll say it (yet)
again:  unless
>> you have very stringent power requirement (or some other unusual 
>> requirement, like very, very low noise),  used (or even new-in-box, 
>> previous generation excess inventory) OEM stuff is far superior to 
>> any build-it-yourself rig you can come up with. So much so, in fact, 
>> that we should really consider the reference recommendation for a ZFS 
>> fileserver to be certain configs of brand-name hardware, and NOT try 
>> to recommend other things to folks.
>>
> Unless you live somewhere with a very small used server market that is!
>
But, I hear there''s this newfangled thingy, called some darned fool 
thing like "the interanets" or some such, that lets you, you know, 
*order* things from far away places using one of those funny PeeCee 
dood-ads, and they like, *deliver* to your door.

And here I was, just getting used to all those nice things from the 
Sears catalog.  Gonna have to learn me a whole new thing again.

Damn kids.

<wink>



Of course, living in certain countries it''s hard to get the hardware 
through customs...

-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA

On Thu, Sep 23, 2010 at 1:08 PM, Dick Hoogendijk <dick at nagual.nl>
wrote:
>  And about what SUN systems are you thinking for ''home
use'' ?
> The likeliness of memory failures might be much higher than becoming a
> millionair, but in the years past I have never had one. And my home sytems
> are rather cheap. Mind you, not the cheapest, but rather cheap. I do buy
> good memory though. So, to me, with a good backup I feel rather safe using
> ZFS. I also had it running for quite some time on a 32bits machine and that
> also worked out fine.
>
We have correctable memory errors on ECC systems on a monthly basis.
It''s
not if they''ll happen but how often.

-- 
Giovanni Tirloni
gtirloni at sysdroid.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100925/d037a90f/attachment.html>

On 09/26/10 07:25 AM, Erik Trimble wrote:
>  On 9/25/2010 1:57 AM, Ian Collins wrote:
>> On 09/25/10 02:54 AM, Erik Trimble wrote:
>>>
>>> Honestly, I''ve said it before, and I''ll say it
(yet) again:  unless
>>> you have very stringent power requirement (or some other unusual 
>>> requirement, like very, very low noise),  used (or even new-in-box,
>>> previous generation excess inventory) OEM stuff is far superior to 
>>> any build-it-yourself rig you can come up with. So much so, in
fact,
>>> that we should really consider the reference recommendation for a 
>>> ZFS fileserver to be certain configs of brand-name hardware, and
NOT
>>> try to recommend other things to folks.
>>>
>> Unless you live somewhere with a very small used server market that is!
>>
>
> But, I hear there''s this newfangled thingy, called some darned
fool
> thing like "the interanets" or some such, that lets you, you
know,
> *order* things from far away places using one of those funny PeeCee 
> dood-ads, and they like, *deliver* to your door.
>
Have you ever had to pay international shipping to the other side of the 
world on a second hand server?!  Not all sellers will ship internationally.

I do bring in a lot of system components, but chassis aren''t worth the
cost.

-- 
Ian.

> Erik Trimble sez:
> Honestly, I''ve said it before, and I''ll say it (yet)
again:  unless you
> have very stringent power requirement (or some other unusual 
> requirement, like very, very low noise),  used (or even new-in-box, 
> previous generation excess inventory) OEM stuff is far superior to any 
> build-it-yourself rig you can come up with. 
It''s horses for courses, I guess. I''ve had to live with server
fan noise and
power requirements and it''s not pleasant. I very much like the
reliability
characteristics of older servers, but they eat a lot of power and are noisy
as you say.

On the other hand, I did the calculation of the difference in cost of 
electricity over two years at my local rates (central Texas) and it''s
easy to
save a few hundred dollars over two years of 24/7 operation with low power
systems. I am NOT necessarily saying that my system is something to emulate, 
nor that my choices are right for everyone, particularly hardware building 
amateurs. My past includes a lot of hardware design and build. So putting 
together a Frankenserver is not something that is daunting. I also have a 
history which includes making educated guesses about failure rates and the 
cost of losing data. So I made choices based on my experience and skills.

For **me**, putting together a server out of commercial parts is a far better 
bet than running a server in a virtual machine on desktop parts of 
any vintage, which was the original question - whether a virtual server on top
of Windows running on some hardware was advisable for a beginner. For me, 
it''s not. The relative merits will vary from user to user according to
their skills
and experience level. I was willing to learn Solaris to get zfs. Given
what''s
happened with Oracle since I started that, that may have been a bad bet, but
my server and data do now live and breathe for better or worse. 

But I have no fears of breathing life into new hardware and copying 
the old data over. Nor is it a trial to me to fire up a last-generation server, 
install a new OS and copy the data over. To me, that''s all a
cost/benefit calculation.
>So much so, in fact, that we   should really consider the reference 
> recommendation for a ZFS fileserver 
> to be certain configs of brand-name hardware, and NOT
> try to recommend other things to folks.
I personally would have loved to have that when I started the zfs/Opensolaris 
trek a year ago. It was not available, and I paid my dues learning the OS and
zfs. I''m not sure, given where Oracle is taking Solaris, that there is
any need to
recommend any particular hardware to folks in general. I think the number of 
people following the path I took, using OpenSolaris to get zfs, and
buying/building
a home machine to do it, are going to nosedive dramatically, by
Oracle''s design.

To me the data stability issues dictated zfs, and Opensolaris was where I
got that. I put up with the labyrinthine mess of figuring out what would and
would
not run OS to get zfs, and it worked OK. To me, data integrity was what I was
after.

I had sub issues. It''s silly (in my estimation) to worry about data
integrity on
disks and not in memory. That made ECC an issue. Hence my burrowing through 
the most cost-efficient way to get ECC. Oh, yeah, cost. I wanted it to be as
cheap as possible, given the other constraints. Then hardware reliability. I 
actually bought an off-duty server locally because of the cost advantages and
the
perceived hardware realiability. I can''t get OS to work on it - yet at
least. I''m sure
that it''s my problems with being an Open Solaris neophyte. But it sure
is noisy.

What **my** compromise was is 
- new hardware to stay inside the shallow end of the failure-rate bathtub
- burn in to get past the infant mortality issues
- ECC as cheaply as possible, given that I actually wanted it to work
- modern SATA controllers for the storage, which dragged in PCIe and compatible
controllers under Opensolaris
- as low a power as possible, as that can save about $100 a year *for me*
- as low a noise factor as possible because I''ve spent too much of my
life listening
to machines desperately try to stay cool.

What I could trade for this was not caring whether the hardware was particularly
fast; it was a layer of data backup, not a mission critical server. And it had
to
run zfs, which is why I started this mess. Also, I don''t have huge data
storage
problems. I enforce the live backup data to be under 4TB. Yep, that''s
tiny by
comparison. I have small problems. 8-)

Result: a new-components server that runs zfs, works on my house network, uses
under 100W as measured at the wall socket, and stores 4TB. I got what I set out
to get, so I''m happy with it. 

This is not the system for everybody, but it works for me. Writing down what 
you''re trying to do is a great tool. People used to get really mad at
me for saying,
in Very Serious Business Meetings "If we were completely successful, what
would
that look like?"  It almost always happened that the guys who were
completely
in agreement that we should do X and not Y disagreed violently when X had to 
be written down.
-- 
This message posted from opensolaris.org

On 25 Sep 2010, at 19:56, Giovanni Tirloni <gtirloni at sysdroid.com>
wrote:
> We have correctable memory errors on ECC systems on a monthly basis.
It''s not if they''ll happen but how often.
"DRAM Errors in the wild: a large-scale field study" is worth a read
if you have time.

http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf

Alex
(@alblue on Twitter)

> 
> 
> On 9/23/2010 at 12:38 PM Erik Trimble wrote:
> 
> | [snip]
> |If you don''t really care about ultra-low-power, then
> there''s
> absolutely 
> |no excuse not to buy a USED server-class machine
> which is 1- or 2- 
> |generations back.  They''re dirt cheap, readily
> available, 
> | [snip]
>  ============> 
> 
> Anyone have a link or two to a place where I can buy
> some dirt-cheap,
> readily available last gen servers?
I would love some links as well. I have heard a lot about "dirt cheap last
gen servers" but nobody ever provides a link.
-- 
This message posted from opensolaris.org

On 9/26/2010 8:06 AM, devsk wrote:
>>
>> On 9/23/2010 at 12:38 PM Erik Trimble wrote:
>>
>> | [snip]
>> |If you don''t really care about ultra-low-power, then
>> there''s
>> absolutely
>> |no excuse not to buy a USED server-class machine
>> which is 1- or 2-
>> |generations back.  They''re dirt cheap, readily
>> available,
>> | [snip]
>>   ============>>
>>
>> Anyone have a link or two to a place where I can buy
>> some dirt-cheap,
>> readily available last gen servers?
> I would love some links as well. I have heard a lot about "dirt cheap
last gen servers" but nobody ever provides a link.
http://www.serversupply.com/products/part_search/pid_lookup.asp?pid=105676
http://www.canvassystems.com/products/c-16-ibm-servers.aspx?_vsrefdom=PPCIBM&gclid=CLTF7NHNpaQCFRpbiAodoSxK5Q&
http://www.glcomp.com/Products/IBM-SystemX-x3500-Server.aspx



Lots, and Lots of stuff from eBay - use them to see which companies are 
in the recycling business, then deal with them directly, rather than 
through eBay.

http://computers.shop.ebay.com/i.html?_nkw=ibm+x3500&_sacat=58058&_sop=12&_dmd=1&_odkw=ibm+x3500&_osacat=0&_trksid=p3286.c0.m270.l1313


Companies specializing in used (often off-lease) business computers:

http://compucycle.net/
http://www.andovercg.com/
http://www.recurrent.com/
http://www.lapkosoft.com/
http://www.weirdstuff.com/
http://synergy.ships2day.com/
http://www.useddell.net/
http://www.vibrant.com/


There''s hordes more.  I''ve dealt with all of the above, and
have no
problems recommending them.



The thing here is that you need to educate yourself *before* going out 
and looking.  You need to spend a non-trivial amount of time reading the 
Support pages for Sun, IBM, HP, and Dell, and be able to either *ask* 
for specific part/model numbers, or be able to interpret what is 
advertised.  The key thing here is that many places will advertised/sell 
you some server, and all the info they have is the model number off the 
front.  If you can understand what this means in terms of hardware, then 
you can get a bang-up deal.

I''ve bought computers from recycling places that were 25% or less of
the
value I could get by *immediately* turning around and selling the system 
somewhere else. All because I could understand the part numbers enough 
to know what I was getting, and the original seller couldn''t (or, in 
most cases, didn''t have the time to bother).  In particular, what is 
usually the best way to get a deal is to look for a machine which has 
(a) very little info about it in the advertisement, other than model 
number, and (b) seems to be noticeably higher in price than what you''ve
seen a "stripped" version of that model go for.   Some of those will
be
stripped systems which the seller doesn''t understand the going rate,
but
many are actually better equipped versions which the additional money is 
more than made up for the significantly better system.

Here''s an example using the IBM x3500:

Model 7977-72y seems to be the most commonly available one right now - 
and the config it''s generally sold in is (2) x dual-core E5140 2.33Ghz 
Xeons, plus 2GB of RAM. No disk, one power supply.   Tends to go for 
$400-500.

I just got a model 7977- F2x, which was advertised as a 2.0Ghz model, 
with nothing else in the ad except the word "loaded".  I paid $750 for
it,  and got a system with (2) quad-core E5335 Xeons, 8GB of RAM, and 
8x73GB SAS drives, plus the redundant power supply.  The extra $300 more 
than covers the cost I would pay for the additional RAM, power supply, 
and drives, and I get twice the CPU core-count for "free".


Be an educated buyer, and the recycled marketplace can be your oyster.  
I''ve actually made enough doing "arbitrage" to cover my costs
of buying
a nice SOHO machine each year.  That is, I can buy and sell 10-12 
systems per year, and make $2000-3000 in profit for not much effort.
I''d
estimate you can sustain a 20-30% profit margin by being a smart 
buyer/seller.  At least on a small scale.


-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA