Zaphod Beeblebrox
2019-Apr-09 21:27 UTC
Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20)
I have a "Ghetto" home RAID array. It's built on compromises and makes use of RAID-Z2 to survive. It consists of two plexes of 8x 4T units of "spinning rust". It's been upgraded and upgraded. It started as 8x 2T, then 8x 2T + 8x 4T then the current 16x 4T. The first 8 disks are connected to motherboard SATA. IIRC, there are 10. Two ports are used for a mirror that it boots from. There's also an SSD in there somhow, so it might be 12 ports on the motherboard. The other 8 disks started life in eSATA port multiplier boxes. That was doubleplusungood, so I got a RAID card based on LSI pulled from a fujitsu server in Japan. That's been upgraded a couple of times... not always a good experience. One problem is that cheap or refurbished drives don't always "like" SAS controllers and FreeBSD. YMMV. Anyways, this is all to introduce the fact that I've seen this behaviour multiple times. You have a drive that leaves the array for some amount of time, and after resilvering, a scrub will find a small amount of bad data. 32 k or 40k or somesuch. In my cranial schema of things, I've chalked it up to out-of-order writing of the drives ... or other such behavior s.t. ZFS doesn't know exactly what has been written. I've often wondered if the fix would be to add an amount of fuzz to the transaction range that is resilvered. On Tue, Apr 9, 2019 at 4:32 PM Karl Denninger <karl at denninger.net> wrote:> On 4/9/2019 15:04, Andriy Gapon wrote: > > On 09/04/2019 22:01, Karl Denninger wrote: > >> the resilver JUST COMPLETED with no errors which means the ENTIRE DISK'S > >> IN USE AREA was examined, compared, and blocks not on the "new member" > >> or changed copied over. > > I think that that's not entirely correct. > > ZFS maintains something called DTL, a dirty-time log, for a missing / > offlined / > > removed device. When the device re-appears and gets resilvered, ZFS > walks only > > those blocks that were born within the TXG range(s) when the device was > missing. > > > > In any case, I do not have an explanation for what you are seeing. > > That implies something much more-serious could be wrong such as given > enough time -- a week, say -- that the DTL marker is incorrect and some > TXGs that were in fact changed since the OFFLINE are not walked through > and synchronized. That would explain why it gets caught by a scrub -- > the resilver is in fact not actually copying all the blocks that got > changed and so when you scrub the blocks are not identical. Assuming > the detached disk is consistent that's not catastrophically bad IF > CAUGHT; where you'd get screwed HARD is in the situation where (for > example) you had a 2-unit mirror, detached one, re-attached it, resilver > says all is well, there is no scrub performed and then the > *non-detached* disk fails before there is a scrub. In that case you > will have permanently destroyed or corrupted data since the other disk > is allegedly consistent but there are blocks *missing* that were never > copied over. > > Again this just showed up on 12.x; it definitely was *not* at issue in > 11.1 at all. I never ran 11.2 in production for a material amount of > time (I went from 11.1 to 12.0 STABLE after the IPv6 fixes were posted > to 12.x) so I don't know if it is in play on 11.2 or not. > > I'll see if it shows up again with 20.00.07.00 card firmware. > > Of note I cannot reproduce this on my test box with EITHER 19.00.00.00 > or 20.00.07.00 firmware when I set up a 3-unit mirror, offline one, make > a crap-ton of changes, offline the second and reattach the third (in > effect mirroring the "take one to the vault" thing) with a couple of > hours elapsed time and a synthetic (e.g. "dd if=/dev/random of=outfile > bs=1m" sort of thing) "make me some new data that has to be resilvered" > workload. I don't know if that's because I need more entropy in the > filesystem than I can reasonably generate this way (e.g. more > fragmentation of files, etc) or whether it's a time-based issue (e.g. > something's wrong with the DTL/TXG thing as you note above in terms of > how it functions and it only happens if the time elapsed causes > something to be subject to a rollover or similar problem.) > > I spent quite a lot of time trying to make reproduce the issue on my > "sandbox" machine and was unable -- and of note it is never a large > quantity of data that is impacted, it's usually only a couple of dozen > checksums that show as bad and fixed. Of note it's also never just one; > if there was a single random hit on a data block due to ordinary bitrot > sort of issues I'd expect only one checksum to be bad. But generating a > realistic synthetic workload over the amount of time involved on a > sandbox is not trivial at all; the system on which this is now happening > handles a lot of email and routine processing of various sorts including > a fair bit of database activity associated with network monitoring and > statistical analysis. > > I'm assuming that using "offline" as a means to do this hasn't become > "invalid" as something that's considered "ok" as a means of doing this > sort of thing.... it certainly has worked perfectly well for a very long > time! > > -- > Karl Denninger > karl at denninger.net <mailto:karl at denninger.net> > /The Market Ticker/ > /[S/MIME encrypted email preferred]/ >
Karl Denninger
2019-Apr-10 01:09 UTC
Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20)
On 4/9/2019 16:27, Zaphod Beeblebrox wrote:> I have a "Ghetto" home RAID array. It's built on compromises and makes use > of RAID-Z2 to survive. It consists of two plexes of 8x 4T units of > "spinning rust". It's been upgraded and upgraded. It started as 8x 2T, > then 8x 2T + 8x 4T then the current 16x 4T. The first 8 disks are > connected to motherboard SATA. IIRC, there are 10. Two ports are used for > a mirror that it boots from. There's also an SSD in there somhow, so it > might be 12 ports on the motherboard. > > The other 8 disks started life in eSATA port multiplier boxes. That was > doubleplusungood, so I got a RAID card based on LSI pulled from a fujitsu > server in Japan. That's been upgraded a couple of times... not always a > good experience. One problem is that cheap or refurbished drives don't > always "like" SAS controllers and FreeBSD. YMMV. > > Anyways, this is all to introduce the fact that I've seen this behaviour > multiple times. You have a drive that leaves the array for some amount of > time, and after resilvering, a scrub will find a small amount of bad data. > 32 k or 40k or somesuch. In my cranial schema of things, I've chalked it > up to out-of-order writing of the drives ... or other such behavior s.t. > ZFS doesn't know exactly what has been written. I've often wondered if the > fix would be to add an amount of fuzz to the transaction range that is > resilvered. > > > On Tue, Apr 9, 2019 at 4:32 PM Karl Denninger <karl at denninger.net> wrote: > >> On 4/9/2019 15:04, Andriy Gapon wrote: >>> On 09/04/2019 22:01, Karl Denninger wrote: >>>> the resilver JUST COMPLETED with no errors which means the ENTIRE DISK'S >>>> IN USE AREA was examined, compared, and blocks not on the "new member" >>>> or changed copied over. >>> I think that that's not entirely correct. >>> ZFS maintains something called DTL, a dirty-time log, for a missing / >> offlined / >>> removed device. When the device re-appears and gets resilvered, ZFS >> walks only >>> those blocks that were born within the TXG range(s) when the device was >> missing. >>> In any case, I do not have an explanation for what you are seeing. >> That implies something much more-serious could be wrong such as given >> enough time -- a week, say -- that the DTL marker is incorrect and some >> TXGs that were in fact changed since the OFFLINE are not walked through >> and synchronized. That would explain why it gets caught by a scrub -- >> the resilver is in fact not actually copying all the blocks that got >> changed and so when you scrub the blocks are not identical. Assuming >> the detached disk is consistent that's not catastrophically bad IF >> CAUGHT; where you'd get screwed HARD is in the situation where (for >> example) you had a 2-unit mirror, detached one, re-attached it, resilver >> says all is well, there is no scrub performed and then the >> *non-detached* disk fails before there is a scrub. In that case you >> will have permanently destroyed or corrupted data since the other disk >> is allegedly consistent but there are blocks *missing* that were never >> copied over. >> >> Again this just showed up on 12.x; it definitely was *not* at issue in >> 11.1 at all. I never ran 11.2 in production for a material amount of >> time (I went from 11.1 to 12.0 STABLE after the IPv6 fixes were posted >> to 12.x) so I don't know if it is in play on 11.2 or not. >> >> I'll see if it shows up again with 20.00.07.00 card firmware. >> >> Of note I cannot reproduce this on my test box with EITHER 19.00.00.00 >> or 20.00.07.00 firmware when I set up a 3-unit mirror, offline one, make >> a crap-ton of changes, offline the second and reattach the third (in >> effect mirroring the "take one to the vault" thing) with a couple of >> hours elapsed time and a synthetic (e.g. "dd if=/dev/random of=outfile >> bs=1m" sort of thing) "make me some new data that has to be resilvered" >> workload. I don't know if that's because I need more entropy in the >> filesystem than I can reasonably generate this way (e.g. more >> fragmentation of files, etc) or whether it's a time-based issue (e.g. >> something's wrong with the DTL/TXG thing as you note above in terms of >> how it functions and it only happens if the time elapsed causes >> something to be subject to a rollover or similar problem.) >> >> I spent quite a lot of time trying to make reproduce the issue on my >> "sandbox" machine and was unable -- and of note it is never a large >> quantity of data that is impacted, it's usually only a couple of dozen >> checksums that show as bad and fixed. Of note it's also never just one; >> if there was a single random hit on a data block due to ordinary bitrot >> sort of issues I'd expect only one checksum to be bad. But generating a >> realistic synthetic workload over the amount of time involved on a >> sandbox is not trivial at all; the system on which this is now happening >> handles a lot of email and routine processing of various sorts including >> a fair bit of database activity associated with network monitoring and >> statistical analysis. >> >> I'm assuming that using "offline" as a means to do this hasn't become >> "invalid" as something that's considered "ok" as a means of doing this >> sort of thing.... it certainly has worked perfectly well for a very long >> time! >> >> -- >> Karl Denninger >> karl at denninger.net <mailto:karl at denninger.net> >> /The Market Ticker/ >> /[S/MIME encrypted email preferred]/The problem with the theory you have put forward Zaphod (which is logical) is that in my *specific* case it shouldn't happen -- which implies (strongly) that there's a code problem somewhere and what you're seeing is not a result of random drive write-reordering. Specifically, I *explicitly* OFFLINE the disk in question, which is a controlled operation and *should* result in a cache flush out of the ZFS code into the drive before it is OFFLINE'd. This should result in the "last written" TXG that the remaining online members have, and the one in the offline member, being consistent. Then I "camcontrol standby" the involved drive, which forces a writeback cache flush and a spindown; in other words, re-ordered or not, the on-platter data *should* be consistent with what the system thinks happened before I yank the physical device. But... it's not. And these are not "ghetto" devices in my case either -- they're all either NAS or enterprise drives and all of them are on a LSI/Avago Sata/SAS adapter -- with no expanders involved (any more anyway; under 11.1 the expander was ENTIRELY reliable, but under 11.2 and 12.0 it's not, so I removed it.)? No "grab whatever" sort of disks are involved in this and I've seen the same behavior across multiple makes and models, which strongly implies it's a not a random firmware bug in some specific model of disk. In the case of a *random* detach I can see it, because the disk that detaches and goes offline might have some number of transactions in the write cache which are lost when the power fails on said device, and thus the committed TXG number is inconsistent with what the remaining online devices have, and write re-ordering could screw you with no way for the code to know it occurred. But in this case the offline was intentional and thus any queued writes should have been flushed down from the OS to the device, and the standby command should flush any on-drive cache to the physical media before the spindle is spun down -- thus there should be no possible way for an out-of-sync "last written" TXG to happen. Unless there's a code problem somewhere.... -- Karl Denninger karl at denninger.net <mailto:karl at denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4897 bytes Desc: S/MIME Cryptographic Signature URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20190409/fb70ee7f/attachment.bin>