thr3ads.net - freebsd stable - ZFS... [May 2019]

If this information is useful, please help other people find it:
Share via:

Michelle Sullivan

2019-May-01 02:26 UTC

ZFS...

Xin LI wrote:>
> On Tue, Apr 30, 2019 at 5:08 PM Michelle Sullivan <michelle at sorbs.net
> <mailto:michelle at sorbs.net>> wrote:
>
>     but in my recent experience 2 issues colliding at the same time
>     results in disaster
>
>
> Do we know exactly what kind of corruption happen to your pool?  If 
> you see it twice in a row, it might suggest a software bug that should 
> be investigated.
>Oh I did spot one interesting bug... though it is benign...


Check out the following (note the difference between 'zpool status' and 
'zpool status -v'):

root at colossus:/mnt # zpool status
   pool: storage
  state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
     continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
   scan: resilver in progress since Mon Apr 29 20:22:03 2019
     6.54T scanned at 0/s, 6.54T issued at 0/s, 28.8T total
     445G resilvered, 22.66% done, no estimated completion time
config:

     NAME        STATE     READ WRITE CKSUM
     storage     ONLINE       0     0     5
       raidz2-0  ONLINE       0     0    20
         mfid11  ONLINE       0     0     0
         mfid10  ONLINE       0     0     0
         mfid8   ONLINE       0     0     0
         mfid7   ONLINE       0     0     0
         mfid0   ONLINE       0     0     0
         mfid5   ONLINE       0     0     0
         mfid4   ONLINE       0     0     0
         mfid3   ONLINE       0     0     0
         mfid2   ONLINE       0     0     0
         mfid14  ONLINE       0     0     0
         mfid15  ONLINE       0     0     0
         mfid6   ONLINE       0     0     0
         mfid9   ONLINE       0     0     0
         mfid13  ONLINE       0     0     0
         mfid1   ONLINE       0     0     0

errors: 4 data errors, use '-v' for a list
root at colossus:/mnt # zpool status
   pool: storage
  state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
     continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
   scan: resilver in progress since Mon Apr 29 20:22:03 2019
     6.54T scanned at 0/s, 6.54T issued at 0/s, 28.8T total
     445G resilvered, 22.66% done, no estimated completion time
config:

     NAME        STATE     READ WRITE CKSUM
     storage     ONLINE       0     0     5
       raidz2-0  ONLINE       0     0    20
         mfid11  ONLINE       0     0     0
         mfid10  ONLINE       0     0     0
         mfid8   ONLINE       0     0     0
         mfid7   ONLINE       0     0     0
         mfid0   ONLINE       0     0     0
         mfid5   ONLINE       0     0     0
         mfid4   ONLINE       0     0     0
         mfid3   ONLINE       0     0     0
         mfid2   ONLINE       0     0     0
         mfid14  ONLINE       0     0     0
         mfid15  ONLINE       0     0     0
         mfid6   ONLINE       0     0     0
         mfid9   ONLINE       0     0     0
         mfid13  ONLINE       0     0     0
         mfid1   ONLINE       0     0     0

errors: 4 data errors, use '-v' for a list
root at colossus:/mnt # zpool status -v
   pool: storage
  state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
     continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
   scan: resilver in progress since Mon Apr 29 20:22:03 2019
     6.54T scanned at 0/s, 6.54T issued at 0/s, 28.8T total
     445G resilvered, 22.66% done, no estimated completion time
config:

     NAME        STATE     READ WRITE CKSUM
     storage     ONLINE       0     0     5
       raidz2-0  ONLINE       0     0    20
         mfid11  ONLINE       0     0     0
         mfid10  ONLINE       0     0     0
         mfid8   ONLINE       0     0     0
         mfid7   ONLINE       0     0     0
         mfid0   ONLINE       0     0     0
         mfid5   ONLINE       0     0     0
         mfid4   ONLINE       0     0     0
         mfid3   ONLINE       0     0     0
         mfid2   ONLINE       0     0     0
         mfid14  ONLINE       0     0     0
         mfid15  ONLINE       0     0     0
         mfid6   ONLINE       0     0     0
         mfid9   ONLINE       0     0     0
         mfid13  ONLINE       0     0     0
         mfid1   ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

         <metadata>:<0x3e>
         <metadata>:<0x5d>
         storage:<0x0>
         storage at now:<0x0>
root at colossus:/mnt # zpool status -v
   pool: storage
  state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
     continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
   scan: resilver in progress since Mon Apr 29 20:22:03 2019
     6.54T scanned at 0/s, 6.54T issued at 0/s, 28.8T total
     445G resilvered, 22.66% done, no estimated completion time
config:

     NAME        STATE     READ WRITE CKSUM
     storage     ONLINE       0     0     7
       raidz2-0  ONLINE       0     0    28
         mfid11  ONLINE       0     0     0
         mfid10  ONLINE       0     0     0
         mfid8   ONLINE       0     0     0
         mfid7   ONLINE       0     0     0
         mfid0   ONLINE       0     0     0
         mfid5   ONLINE       0     0     0
         mfid4   ONLINE       0     0     0
         mfid3   ONLINE       0     0     0
         mfid2   ONLINE       0     0     0
         mfid14  ONLINE       0     0     0
         mfid15  ONLINE       0     0     0
         mfid6   ONLINE       0     0     0
         mfid9   ONLINE       0     0     0
         mfid13  ONLINE       0     0     0
         mfid1   ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

         <metadata>:<0x3e>
         <metadata>:<0x5d>
         storage:<0x0>
         storage at now:<0x0>
root at colossus:/mnt # zpool status -v
   pool: storage
  state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
     continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
   scan: resilver in progress since Mon Apr 29 20:22:03 2019
     6.54T scanned at 0/s, 6.54T issued at 0/s, 28.8T total
     445G resilvered, 22.66% done, no estimated completion time
config:

     NAME        STATE     READ WRITE CKSUM
         mfid10  ONLINE       0     0     0
         mfid8   ONLINE       0     0     0
         mfid7   ONLINE       0     0     0
         mfid0   ONLINE       0     0     0
         mfid5   ONLINE       0     0     0
         mfid4   ONLINE       0     0     0
         mfid3   ONLINE       0     0     0
         mfid2   ONLINE       0     0     0
         mfid14  ONLINE       0     0     0
         mfid15  ONLINE       0     0     0
         mfid6   ONLINE       0     0     0
         mfid9   ONLINE       0     0     0
         mfid13  ONLINE       0     0     0
         mfid1   ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

         <metadata>:<0x3e>
         <metadata>:<0x5d>
         storage:<0x0>
         storage at now:<0x0>
root at colossus:/mnt # zpool status -v
   pool: storage
  state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
     continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
   scan: resilver in progress since Mon Apr 29 20:22:03 2019
     6.54T scanned at 0/s, 6.54T issued at 0/s, 28.8T total
     445G resilvered, 22.66% done, no estimated completion time
config:

     NAME        STATE     READ WRITE CKSUM
     storage     ONLINE       0     0    11
       raidz2-0  ONLINE       0     0    44
         mfid11  ONLINE       0     0     0
         mfid10  ONLINE       0     0     0
         mfid8   ONLINE       0     0     0
         mfid7   ONLINE       0     0     0
         mfid0   ONLINE       0     0     0
         mfid5   ONLINE       0     0     0
         mfid4   ONLINE       0     0     0
         mfid3   ONLINE       0     0     0
         mfid2   ONLINE       0     0     0
         mfid14  ONLINE       0     0     0
         mfid15  ONLINE       0     0     0
         mfid6   ONLINE       0     0     0
         mfid9   ONLINE       0     0     0
         mfid13  ONLINE       0     0     0
         mfid1   ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

         <metadata>:<0x3e>
         <metadata>:<0x5d>
         storage:<0x0>
         storage at now:<0x0>
root at colossus:/mnt #


-- 
Michelle Sullivan
http://www.mhix.org/

Borja Marcos

2019-May-03 07:09 UTC

head link

ZFS...

> On 1 May 2019, at 04:26, Michelle Sullivan <michelle at sorbs.net>
wrote:
> 
>        mfid8   ONLINE       0     0     0
Anyway I think this is a mistake (mfid). I know, HBA makers have been insisting
on having their firmware getting in the middle,
which is a bad thing.

The right way to use disks is to give ZFS access to the plain CAM devices, not
thorugh some so-called JBOD on a RAID
controller which, at least for a long time, has been a *logical* ?RAID0? volume
on a single disk. That additional layer can
completely break the semantics of transaction writes and cache flushes. 

With some older cards it can be tricky to achieve, from patching source drivers
to enabling a sysctl tunable or even
flashing the card to turn it into a plain HBA with no RAID features (or minimal
ones).

If your drives are not called /dev/daX or /dev/adaX you are likely to be in
trouble. Unless something has really changed recently
you don?t want ?mfid? or ?mfisyspd?.

I have suffered hidden data corruption due to a faulty HBA and failures of old
disks, and in all cases ZFS has survived brilianty.

And actually ZFS works on somewhat unreliable hardware. The problem is not
non-perfect hardware, but *evil* hardware with
firmware based on some assumptions that won?t work with ZFS. 

But I agree, non-ECC memory can be a problem. In my case all of the servers had
ECC.

Borja.

freebsd stable - May 2019 - ZFS...

ZFS...

ZFS...