zfs discuss - Jul 2010 - Cache flush (or the lack of such) and corruption

Hi all

I''ve been reading a lot of messages on this list about potential and
actual corruption of a zpool due to cache flush problems and whatnot, and I find
myself amazed.

I just wonder how a zpool compares with a good old filesystem when it comes to
filesystem errors. It seems several of the members of this list have encountered
problems where they had to boot a live CD to get their pool back, whereas a
normal filesystem won''t give this problem. The old-time filesystem
might have corrupted data, but it still gets up.

Can someone give me some good input on this, and perhaps how to avoid an enitre
pool to become unavailable?

PS: I''m using small RAIDz2 pools with sufficient amount of redundancy

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
roy at karlsbakk.net
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er
et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av
idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og
relevante synonymer p? norsk.

On Jul 10, 2010, at 12:53 PM, Roy Sigurd Karlsbakk
wrote:
> Hi all
> 
> I''ve been reading a lot of messages on this list about potential
and actual corruption of a zpool due to cache flush problems and whatnot, and I
find myself amazed.
Why are you amazed?  Storage devices have been losing data since they
were invented :-P
> I just wonder how a zpool compares with a good old filesystem when it comes
to filesystem errors. It seems several of the members of this list have
encountered problems where they had to boot a live CD to get their pool back,
whereas a normal filesystem won''t give this problem. The old-time
filesystem might have corrupted data, but it still gets up.
Depends on the failure mode.  I''ve spent hundreds (thousands?) of hours
attempting to recover data from backup tape because of bad hardware, firmware,
and file systems. The major difference is that ZFS cares that the data is not
correct, while older file systems did not care about the data. 

[no, fsck does not correct data errors]
> Can someone give me some good input on this, and perhaps how to avoid an
enitre pool to become unavailable?
Use good quality hardware and redundancy.
> PS: I''m using small RAIDz2 pools with sufficient amount of
redundancy
Good.
 -- richard

-- 
Richard Elling
richard at nexenta.com   +1-760-896-4422
ZFS and NexentaStor training, Rotterdam, July 13-15, 2010
http://nexenta-rotterdam.eventbrite.com/

----- Original Message -----
> Depends on the failure mode. I''ve spent hundreds (thousands?) of
hours
> attempting to recover data from backup tape because of bad hardware,
> firmware,
> and file systems. The major difference is that ZFS cares that the data
> is not
> correct, while older file systems did not care about the data.
It still seems like ZFS has a problem with its metadata. Reports of loss of
pools because of metadata errors is what is worrying me. Can you give me any
input on how to avoid this?

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
roy at karlsbakk.net
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er
et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av
idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og
relevante synonymer p? norsk.

On Sat, 10 Jul 2010, Roy Sigurd Karlsbakk wrote:
> I''ve been reading a lot of messages on this list about potential
and
> actual corruption of a zpool due to cache flush problems and 
> whatnot, and I find myself amazed.
You should not be amazed.  People only take their radio in for repair 
when it breaks.  Therefore, the radio repair man only experiences 
broken radios.  Meanwhile, millions of radios continue to work without 
fail.  It is the same on discussion lists and forums.
> I just wonder how a zpool compares with a good old filesystem when 
> it comes to filesystem errors. It seems several of the members of 
> this list have encountered problems where they had to boot a live CD 
> to get their pool back, whereas a normal filesystem won''t give
this
> problem. The old-time filesystem might have corrupted data, but it 
> still gets up.
Most "old-time filesystems" are tremendously smaller than
today''s zfs
storage pools, and they might even be on just one disk.  Regardless, 
only someone with severely failing memory might think that "old-time 
filesystems" are somehow less failure prone than a zfs storage pool. 
The "good old days" does not apply to filesystems.

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On 10-Jul-10, at 4:57 PM, Roy Sigurd Karlsbakk wrote:
> ----- Original Message -----
>> Depends on the failure mode. I''ve spent hundreds (thousands?)
of
>> hours
>> attempting to recover data from backup tape because of bad hardware,
>> firmware,
>> and file systems. The major difference is that ZFS cares that the  
>> data
>> is not
>> correct, while older file systems did not care about the data.
>
> It still seems like ZFS has a problem with its metadata. Reports of  
> loss of pools because of metadata errors is what is worrying me. Can  
> you give me any input on how to avoid this?
Roy

It needs to be pointed out that this only causes an integrity problem  
for zfs *when the hardware stack is faulty* (not respecting flush).  
And it obviously then affects all systems which assume working  
barriers (RDBMS, reiser3fs, ext3fs, other journaling systems, etc).

--Toby
>
> Vennlige hilsener / Best regards
>
> roy
> --
> Roy Sigurd Karlsbakk
> (+47) 97542685
> roy at karlsbakk.net
> http://blogg.karlsbakk.net/
> --
> I all pedagogikk er det essensielt at pensum presenteres  
> intelligibelt. Det er et element?rt imperativ for alle pedagoger ?  
> unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de  
> fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk.
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

ZFS is a copy-on-write filesystem.  The important point is that if a single byte
in a file is changed then the containing block is rewritten elsewhere, requiring
that the file block pointers be rewritten - and when these are rewritten they
are likewise written elsewhere and pointers to *them* need to be rewritten,
"recursively" all the way to the root of the filesystem, or ?berblock.

In other words, a write anywhere necessitates changes in high-level filesystem
metadata.  In an archaic filesystem such as NTFS changes are local: write a file
require little metadata be changed and rarely metadata for the entire
filesystem.

As a result ZFS *is* more prone to pool loss *when the hardware screws up* since
filesystem metadata is written more often.

At one time there was talk of ZFS implementing a "deferred
reallocation" scheme in ?berblock updates.  The would greatly improve
ZFS'' ability to withstand poorly designed hardware.

PS RAIDZ2 is a good thing but is mostly irrelevant to pool corruption.  You need
backups.  I set up all client installations with a hot backup via zfs send/recv.
-- 
This message posted from opensolaris.org