zfs discuss - Jan 2009 - ZFS disk read failure when pools are simultaneously scrubbed, x86 snv_104

Hi.

Running snv_104 x86 against some very generic hardware as a testbed for some fun
projects and as a home fileserver. Rough specifications of the host:

* Intel Q6600
* 6GB DDR2
* Multiple 250GB, 500GB SATA connected HDD''s of mixed vendors
* Gigabyte GA-DQ6 series motherboard
* etc.

The problem or interesting scenario.

I decided to cron a zpool scrub of all three zpool''s on the host system
simultaneously. Something like this:

00 01 * * * /usr/sbin/zpool scrub backups > /dev/null 2>&1
00 01 * * * /usr/sbin/zpool scrub ztank > /dev/null 2>&1
00 01 * * * /usr/sbin/zpool scrub zebraware_root > /dev/null 2>&1

So, we know from reading documentation that:

[i]Because  scrubbing  and  resilvering  are  I/O-intensive
 operations, ZFS only allows one at a time. If a scrub is
 already in progress,  the  "zpool  scrub"  command  ter-
 minates  it  and starts a new scrub. If a resilver is in
 progress, ZFS does not allow a scrub to be started until
 the resilver completes[/i]

Please note the "[b]ZFS only allows one at a time[/b]" statement.
Maybe relevant to what I''m about to explain. Maybe not.

I''ve noticed that when I lay my cron out in such a way two things
happen:

1. On the "backups" pool, which is a simple zpool "stripe"
with no redundancy, mirroring or anything of use, the pool will fault at some
interminant point inside the scrub operation

2. The same thing will ALSO occur on the root pool (zebraware_root).

However, if the scrubs are cron''ed at DIFFERENT times, allowing a
period of time to lapse where each will complete before the next starts, these
errors are not presented in /var/adm/messages, and a "zpool status -x"
reports all pools as healthy. It is only if the pools are cron''ed to
scrub simultaneously will read errors occur. Some interesting output, occurring
just after the simultaneous scrub starts on the three pools that exist on the
host:

Dec 30 06:37:22 rapoosev5 scsi: [ID 107833 kern.warning] WARNING: /pci at
0,0/pci8086,2948 at 1c,4/pci-ide at 0/ide at 0 (ata0):
Dec 30 06:37:22 rapoosev5       timeout: abort request, target=0 lun=0
Dec 30 06:37:22 rapoosev5 scsi: [ID 107833 kern.warning] WARNING: /pci at
0,0/pci8086,2948 at 1c,4/pci-ide at 0/ide at 0 (ata0):
Dec 30 06:37:22 rapoosev5       timeout: abort device, target=0 lun=0
Dec 30 06:37:22 rapoosev5 scsi: [ID 107833 kern.warning] WARNING: /pci at
0,0/pci8086,2948 at 1c,4/pci-ide at 0/ide at 0 (ata0):
Dec 30 06:37:22 rapoosev5       timeout: reset target, target=0 lun=0
Dec 30 06:37:22 rapoosev5 scsi: [ID 107833 kern.warning] WARNING: /pci at
0,0/pci8086,2948 at 1c,4/pci-ide at 0/ide at 0 (ata0):
Dec 30 06:37:22 rapoosev5       timeout: reset bus, target=0 lun=0
Dec 30 06:37:22 rapoosev5 scsi: [ID 107833 kern.warning] WARNING: /pci at
0,0/pci8086,2948 at 1c,4/pci-ide at 0/ide at 0 (ata0):
Dec 30 06:37:22 rapoosev5       timeout: early timeout, target=1 lun=0
Dec 30 06:37:22 rapoosev5 scsi: [ID 107833 kern.warning] WARNING: /pci at
0,0/pci8086,2948 at 1c,4/pci-ide at 0/ide at 0 (ata0):
Dec 30 06:37:22 rapoosev5       timeout: early timeout, target=0 lun=0
Dec 30 06:37:22 rapoosev5 gda: [ID 107833 kern.warning] WARNING: /pci at
0,0/pci8086,2948 at 1c,4/pci-ide at 0/ide at 0/cmdk at 0,0 (Disk0):
Dec 30 06:37:22 rapoosev5       Error for command ''read
sector'' Error Level: Informational
Dec 30 06:37:22 rapoosev5 gda: [ID 107833 kern.notice]  Sense Key: aborted
command
Dec 30 06:37:22 rapoosev5 gda: [ID 107833 kern.notice]  Vendor ''Gen-ATA
'' error code: 0x3
Dec 30 06:37:22 rapoosev5 gda: [ID 107833 kern.warning] WARNING: /pci at
0,0/pci8086,2948 at 1c,4/pci-ide at 0/ide at 0/cmdk at 1,0 (Disk1):
Dec 30 06:37:22 rapoosev5       Error for command ''read
sector'' Error Level: Informational
Dec 30 06:37:22 rapoosev5 gda: [ID 107833 kern.notice]  Sense Key: aborted
command
Dec 30 06:37:22 rapoosev5 gda: [ID 107833 kern.notice]  Vendor ''Gen-ATA
'' error code: 0x3
Dec 30 06:37:22 rapoosev5 gda: [ID 107833 kern.warning] WARNING: /pci at
0,0/pci8086,2948 at 1c,4/pci-ide at 0/ide at 0/cmdk at 0,0 (Disk0):
Dec 30 06:37:22 rapoosev5       Error for command ''read
sector'' Error Level: Informational
Dec 30 06:37:22 rapoosev5 gda: [ID 107833 kern.notice]  Sense Key: aborted
command
Dec 30 06:37:22 rapoosev5 gda: [ID 107833 kern.notice]  Vendor ''Gen-ATA
'' error code: 0x3

Shortly after this, we''ll see:

Jan  1 06:39:58 rapoosev5 fmd: [ID 441519 daemon.error] SUNW-MSG-ID:
ZFS-8000-FD, TYPE: Fault, VER: 1, SEVERITY: Major
Jan  1 06:39:58 rapoosev5 EVENT-TIME: Thu Jan  1 06:39:56 EST 2009
Jan  1 06:39:58 rapoosev5 PLATFORM: P35-DQ6, CSN:  , HOSTNAME: rapoosev5
Jan  1 06:39:58 rapoosev5 SOURCE: zfs-diagnosis, REV: 1.0
Jan  1 06:39:58 rapoosev5 EVENT-ID: e6d95684-5ec0-4897-d761-b7e16ed40f2c
Jan  1 06:39:58 rapoosev5 DESC: The number of I/O errors associated with a ZFS
device exceeded
Jan  1 06:39:58 rapoosev5            acceptable levels.  Refer to
http://sun.com/msg/ZFS-8000-FD for more information.

And bang. Part of a pool is taken offline. We all know where that ends up. At
this point, I can issue a "zpool clear" to the filesystems in
question, and the pool clears, comes back online without any issues at all.
Further to this, it is only ever READ ERRORS in a "zpool status"
output that shows up. Never write errors, nor checksum validation problems.

So, my puzzling thoughts:

1. Am I just experiencing some form of crappy consumer grade controller I/O
limitations or an issue of the controllers on this consumer grade kit not being
up to the task of handling multiple scrubs occurring on different filesystems at
any given time>?

2. Is this natural and to be expected (and moreover, am I breaking the rules) by
attempting to scrub more than one pool at once - ergo [i]"well, what did
you expect?[/i]"

Out of fear and sensibility, I''ve never simultaneously scrubbed
production pools on our 6 series arrays at work, or for anything that actually
matters - but I am interested in getting to the bottom of this, all the same.

Thanks!

z
-- 
This message posted from opensolaris.org

On 03 January, 2009 - Jake Carroll sent me these 5,9K bytes:
> Hi.
> 
> Running snv_104 x86 against some very generic hardware as a testbed for
some fun projects and as a home fileserver. Rough specifications of the host:
> 
> * Intel Q6600
> * 6GB DDR2
> * Multiple 250GB, 500GB SATA connected HDD''s of mixed vendors
> * Gigabyte GA-DQ6 series motherboard
> * etc.
> 
> The problem or interesting scenario.
> 
> I decided to cron a zpool scrub of all three zpool''s on the host
system simultaneously. Something like this:
> 
> 00 01 * * * /usr/sbin/zpool scrub backups > /dev/null 2>&1
> 00 01 * * * /usr/sbin/zpool scrub ztank > /dev/null 2>&1
> 00 01 * * * /usr/sbin/zpool scrub zebraware_root > /dev/null 2>&1
> 
> So, we know from reading documentation that:
> 
> [i]Because  scrubbing  and  resilvering  are  I/O-intensive
>  operations, ZFS only allows one at a time. If a scrub is
>  already in progress,  the  "zpool  scrub"  command  ter-
>  minates  it  and starts a new scrub. If a resilver is in
>  progress, ZFS does not allow a scrub to be started until
>  the resilver completes[/i]
> 
> Please note the "[b]ZFS only allows one at a time[/b]" statement.
> Maybe relevant to what I''m about to explain. Maybe not.
One at a time per pool.
> So, my puzzling thoughts:
> 
> 1. Am I just experiencing some form of crappy consumer grade
> controller I/O limitations or an issue of the controllers on this
> consumer grade kit not being up to the task of handling multiple
> scrubs occurring on different filesystems at any given time>?
Probably. When getting too much load, you might get some overhearing or
something. Shouldn''t happen if the hw + drivers works as expected..

You will probably get similar results if you start too much regular io..
> 2. Is this natural and to be expected (and moreover, am I breaking the
> rules) by attempting to scrub more than one pool at once - ergo
> [i]"well, what did you expect?[/i]"
I scrub more than one pool at a time all the time, no issues.
> Out of fear and sensibility, I''ve never simultaneously scrubbed
> production pools on our 6 series arrays at work, or for anything that
> actually matters - but I am interested in getting to the bottom of
> this, all the same.
/Tomas
-- 
Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of Ume?
`- Sysadmin at {cs,acc}.umu.se

On Sat, 3 Jan 2009, Jake Carroll wrote:
>
> 1. Am I just experiencing some form of crappy consumer grade 
> controller I/O limitations or an issue of the controllers on this 
> consumer grade kit not being up to the task of handling multiple 
> scrubs occurring on different filesystems at any given time>?
This seems like the result of either a device weakness, or a device 
driver weakness.  Many people here have requested shorter timeouts on 
devices and it seems that this is an example of why shorter timeouts 
are not a good idea.

It would useful to find out of you can cause this problem by causing 
severe I/O loads other than via ''scrub''.

Bob
=====================================Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/