Stephan Budach
2012-Sep-20 07:45 UTC
[zfs-discuss] Data corruption but no faulted drive/vdev
Hi,
a couple of days we had an issue with one of our FC switches which led
to a switch restart. Due to this issue the zpool vdevs had been
disconnected from the Solaris 11 host, which had to be restarted to get
things running again.
Now, when I imported the zpools for this host, at the first glance,
everything looked good: zpool list didn''t show any issues, but zpool
status did. It claims some data corruption and I seem to cannot get rid
of if.
This is what zpool status says about this zpool:
zpool status -v VSM_POOL_tr1204631_32
pool: VSM_POOL_tr1204631_32
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://www.sun.com/msg/ZFS-8000-8A
scan: scrub in progress since Wed Sep 19 12:26:30 2012
4,63T scanned out of 13,9T at 64,3M/s, 42h11m to go
0 repaired, 33,20% done
config:
NAME STATE READ WRITE CKSUM
VSM_POOL_tr1204631_32 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t2023001378E0B118d0 ONLINE 0 0 0
c0t2017001378E09A18d0 ONLINE 0 0 0
c0t2057001378E07918d0 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
c0t2024001378E0B118d0 ONLINE 0 0 0
c0t2018001378E09A18d0 ONLINE 0 0 0
c0t2058001378E07918d0 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
c0t2025001378E0B118d0 ONLINE 0 0 0
c0t2019001378E09A18d0 ONLINE 0 0 0
c0t2059001378E07918d0 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
c0t2013001378E09A18d0 ONLINE 0 0 0
c0t2020001378E0B118d0 ONLINE 0 0 0
c0t2068001378E07918d0 ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
c0t2016001378E09A18d0 ONLINE 0 0 0
c0t2022001378E0B118d0 ONLINE 0 0 0
c0t205B001378E07918d0 ONLINE 0 0 0
mirror-5 ONLINE 0 0 0
c0t2021001378E0B118d0 ONLINE 0 0 0
c0t2015001378E09A18d0 ONLINE 0 0 0
c0t205C001378E07918d0 ONLINE 0 0 0
mirror-6 ONLINE 0 0 0
c0t2026001378E0B118d0 ONLINE 0 0 0
c0t201A001378E09A18d0 ONLINE 0 0 0
c0t205D001378E07918d0 ONLINE 0 0 0
mirror-7 ONLINE 0 0 0
c0t2027001378E0B118d0 ONLINE 0 0 0
c0t201B001378E09A18d0 ONLINE 0 0 0
c0t205E001378E07918d0 ONLINE 0 0 0
mirror-8 ONLINE 0 0 0
c0t201C001378E09A18d0 ONLINE 0 0 0
c0t2028001378E0B118d0 ONLINE 0 0 0
c0t205F001378E07918d0 ONLINE 0 0 0
mirror-9 ONLINE 0 0 0
c0t201D001378E09A18d0 ONLINE 0 0 0
c0t2029001378E0B118d0 ONLINE 0 0 0
c0t2060001378E07918d0 ONLINE 0 0 0
mirror-10 ONLINE 0 0 0
c0t201E001378E09A18d0 ONLINE 0 0 0
c0t202A001378E0B118d0 ONLINE 0 0 0
c0t2061001378E07918d0 ONLINE 0 0 0
mirror-11 ONLINE 0 0 0
c0t201F001378E09A18d0 ONLINE 0 0 0
c0t202B001378E0B118d0 ONLINE 0 0 0
c0t2064001378E07918d0 ONLINE 0 0 0
mirror-12 ONLINE 0 0 0
c0t2020001378E09A18d0 ONLINE 0 0 0
c0t202C001378E0B118d0 ONLINE 0 0 0
c0t2063001378E07918d0 ONLINE 0 0 0
mirror-13 ONLINE 0 0 0
c0t2021001378E09A18d0 ONLINE 0 0 0
c0t202D001378E0B118d0 ONLINE 0 0 0
c0t2065001378E07918d0 ONLINE 0 0 0
mirror-14 ONLINE 0 0 0
c0t2022001378E09A18d0 ONLINE 0 0 0
c0t202E001378E0B118d0 ONLINE 0 0 0
c0t2066001378E07918d0 ONLINE 0 0 0
spares
c0t2023001378E09A18d0 AVAIL
c0t202F001378E0B118d0 AVAIL
c0t2067001378E07918d0 AVAIL
errors: Permanent errors have been detected in the following files:
/VSM_POOL_tr1204631_32/FCE-Test/
VSM_POOL_tr1204631_32/VSM_POOL_01:<0x0>
Now, /VSM_POOL_tr1204631_32/FCE-Test/ is the mount point of a zfs volume
on that zpool and I can''t find anything problematic with the zfs
volume.
Since I don''t need that zfs volume anymore, I could easily remove it.
The second error, however, disturbs me - it doesn''t seem to point to a
file or directory.
Thanks,
budy
--
Stephan Budach
Jung von Matt/it-services GmbH
Glash?ttenstra?e 79
20357 Hamburg
Tel: +49 40-4321-1353
Fax: +49 40-4321-1114
E-Mail: stephan.budach at jvm.de
Internet: http://www.jvm.com
Gesch?ftsf?hrer: Frank Wilhelm, Stephan Budach (stellv.)
AG HH HRB 98380
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120920/0ba8eea9/attachment.html>
Sašo Kiselkov
2012-Sep-20 08:31 UTC
[zfs-discuss] Data corruption but no faulted drive/vdev
Have you tried a zpool clear and subsequent scrub to see if the error pops up again? Cheers, -- Saso On 09/20/2012 09:45 AM, Stephan Budach wrote:> Hi, > > a couple of days we had an issue with one of our FC switches which led > to a switch restart. Due to this issue the zpool vdevs had been > disconnected from the Solaris 11 host, which had to be restarted to get > things running again. > > Now, when I imported the zpools for this host, at the first glance, > everything looked good: zpool list didn''t show any issues, but zpool > status did. It claims some data corruption and I seem to cannot get rid > of if. > > This is what zpool status says about this zpool: > > zpool status -v VSM_POOL_tr1204631_32 > pool: VSM_POOL_tr1204631_32 > state: ONLINE > status: One or more devices has experienced an error resulting in data > corruption. Applications may be affected. > action: Restore the file in question if possible. Otherwise restore the > entire pool from backup. > see: http://www.sun.com/msg/ZFS-8000-8A > scan: scrub in progress since Wed Sep 19 12:26:30 2012 > 4,63T scanned out of 13,9T at 64,3M/s, 42h11m to go > 0 repaired, 33,20% done > config: > > NAME STATE READ WRITE CKSUM > VSM_POOL_tr1204631_32 ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > c0t2023001378E0B118d0 ONLINE 0 0 0 > c0t2017001378E09A18d0 ONLINE 0 0 0 > c0t2057001378E07918d0 ONLINE 0 0 0 > mirror-1 ONLINE 0 0 0 > c0t2024001378E0B118d0 ONLINE 0 0 0 > c0t2018001378E09A18d0 ONLINE 0 0 0 > c0t2058001378E07918d0 ONLINE 0 0 0 > mirror-2 ONLINE 0 0 0 > c0t2025001378E0B118d0 ONLINE 0 0 0 > c0t2019001378E09A18d0 ONLINE 0 0 0 > c0t2059001378E07918d0 ONLINE 0 0 0 > mirror-3 ONLINE 0 0 0 > c0t2013001378E09A18d0 ONLINE 0 0 0 > c0t2020001378E0B118d0 ONLINE 0 0 0 > c0t2068001378E07918d0 ONLINE 0 0 0 > mirror-4 ONLINE 0 0 0 > c0t2016001378E09A18d0 ONLINE 0 0 0 > c0t2022001378E0B118d0 ONLINE 0 0 0 > c0t205B001378E07918d0 ONLINE 0 0 0 > mirror-5 ONLINE 0 0 0 > c0t2021001378E0B118d0 ONLINE 0 0 0 > c0t2015001378E09A18d0 ONLINE 0 0 0 > c0t205C001378E07918d0 ONLINE 0 0 0 > mirror-6 ONLINE 0 0 0 > c0t2026001378E0B118d0 ONLINE 0 0 0 > c0t201A001378E09A18d0 ONLINE 0 0 0 > c0t205D001378E07918d0 ONLINE 0 0 0 > mirror-7 ONLINE 0 0 0 > c0t2027001378E0B118d0 ONLINE 0 0 0 > c0t201B001378E09A18d0 ONLINE 0 0 0 > c0t205E001378E07918d0 ONLINE 0 0 0 > mirror-8 ONLINE 0 0 0 > c0t201C001378E09A18d0 ONLINE 0 0 0 > c0t2028001378E0B118d0 ONLINE 0 0 0 > c0t205F001378E07918d0 ONLINE 0 0 0 > mirror-9 ONLINE 0 0 0 > c0t201D001378E09A18d0 ONLINE 0 0 0 > c0t2029001378E0B118d0 ONLINE 0 0 0 > c0t2060001378E07918d0 ONLINE 0 0 0 > mirror-10 ONLINE 0 0 0 > c0t201E001378E09A18d0 ONLINE 0 0 0 > c0t202A001378E0B118d0 ONLINE 0 0 0 > c0t2061001378E07918d0 ONLINE 0 0 0 > mirror-11 ONLINE 0 0 0 > c0t201F001378E09A18d0 ONLINE 0 0 0 > c0t202B001378E0B118d0 ONLINE 0 0 0 > c0t2064001378E07918d0 ONLINE 0 0 0 > mirror-12 ONLINE 0 0 0 > c0t2020001378E09A18d0 ONLINE 0 0 0 > c0t202C001378E0B118d0 ONLINE 0 0 0 > c0t2063001378E07918d0 ONLINE 0 0 0 > mirror-13 ONLINE 0 0 0 > c0t2021001378E09A18d0 ONLINE 0 0 0 > c0t202D001378E0B118d0 ONLINE 0 0 0 > c0t2065001378E07918d0 ONLINE 0 0 0 > mirror-14 ONLINE 0 0 0 > c0t2022001378E09A18d0 ONLINE 0 0 0 > c0t202E001378E0B118d0 ONLINE 0 0 0 > c0t2066001378E07918d0 ONLINE 0 0 0 > spares > c0t2023001378E09A18d0 AVAIL > c0t202F001378E0B118d0 AVAIL > c0t2067001378E07918d0 AVAIL > > errors: Permanent errors have been detected in the following files: > > /VSM_POOL_tr1204631_32/FCE-Test/ > VSM_POOL_tr1204631_32/VSM_POOL_01:<0x0> > > > Now, /VSM_POOL_tr1204631_32/FCE-Test/ is the mount point of a zfs volume > on that zpool and I can''t find anything problematic with the zfs volume. > Since I don''t need that zfs volume anymore, I could easily remove it. > The second error, however, disturbs me - it doesn''t seem to point to a > file or directory. > > Thanks, > budy > > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss