zfs discuss - Oct 2009 - scrubing/resilvering - controller problem

Hi,

We are running b118, with a LSI 3801 controller which is connected to 44 drives
(yes it''s a lot behind a single controller). We also use a pair of ssd
connected to another controller for read cache.
Everything works fine and we achieve acceptable performance for our needs.
However, during scrubbing or resilvering operations, it seems ZFS generates so
much traffic that it overwhelmes the controller. The controller then logs the
following errors:

Oct  6 07:30:04 nas101 scsi: [ID 107833 kern.warning] WARNING: /pci at
0,0/pci8086,65fa at 4/pci1000,30a0 at 0/sd at 16,0 (sd19):
Oct  6 07:30:04 nas101        incomplete read- retrying


Is there anything that can be done to slow down zfs operations such as
resilvering/scrubbing? We tried tuning  zfs:zfs_vdev_max_pending but it did not
really help.
This is a bit frustrating because this configuration works well to serve data.
It''s just too aggressive when the kernel accesses drives for some
operations.


Iostat looks liks this:
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b s/w h/w trn tot
device
 9681.5   37.3 116403.3   35.1  0.0 30.2    0.0    3.1   0 1000   0  21   8  29
c9
  420.3    1.5 5058.2    1.4  0.0  1.3    0.0    3.1   0  44   0   0   0   0
c9t8d0



Any help would be appreciated.

thanks,

JJ
-- 
This message posted from opensolaris.org

You might try setting zfs_scrub_limit to 1 or 2 and attach a customer  
service record to :

	6494473 ZFS needs a way to slow down resilvering

-r


Le 7 oct. 09 ? 06:14, John a ?crit :
> Hi,
>
> We are running b118, with a LSI 3801 controller which is connected  
> to 44 drives (yes it''s a lot behind a single controller). We also
> use a pair of ssd connected to another controller for read cache.
> Everything works fine and we achieve acceptable performance for our  
> needs.
> However, during scrubbing or resilvering operations, it seems ZFS  
> generates so much traffic that it overwhelmes the controller. The  
> controller then logs the following errors:
>
> Oct  6 07:30:04 nas101 scsi: [ID 107833 kern.warning] WARNING: / 
> pci at 0,0/pci8086,65fa at 4/pci1000,30a0 at 0/sd at 16,0 (sd19):
> Oct  6 07:30:04 nas101        incomplete read- retrying
>
>
> Is there anything that can be done to slow down zfs operations such  
> as resilvering/scrubbing? We tried tuning  zfs:zfs_vdev_max_pending  
> but it did not really help.
> This is a bit frustrating because this configuration works well to  
> serve data. It''s just too aggressive when the kernel accesses
drives
> for some operations.
>
>
> Iostat looks liks this:
>    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b s/w h/w  
> trn tot device
> 9681.5   37.3 116403.3   35.1  0.0 30.2    0.0    3.1   0 1000   0   
> 21   8  29 c9
>  420.3    1.5 5058.2    1.4  0.0  1.3    0.0    3.1   0  44   0    
> 0   0   0 c9t8d0
>
>
>
> Any help would be appreciated.
>
> thanks,
>
> JJ
> -- 
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2431 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091008/b7a27bc5/attachment.bin>