Demian Phillips
2010-May-21 21:32 UTC
[zfs-discuss] ZFS no longer working with FC devices.
For years I have been running a zpool using a Fibre Channel array with
no problems. I would scrub every so often and dump huge amounts of
data (tens or hundreds of GB) around and it never had a problem
outside of one confirmed (by the array) disk failure.
I upgraded to sol10x86 05/09 last year and since then I have
discovered any sufficiently high I/O from ZFS starts causing timeouts
and off-lining disks. This leads to failure (once rebooted and cleaned
all is well) long term because you can no longer scrub reliably.
ATA, SATA and SAS do not seem to suffer this problem.
I tried upgrading, and then doing a fresh load of U8 and the problem persists.
My FC hardware is:
Sun A5100 (14 disk) array.
Hitachi 146GB FC disks (started with 9GB SUN disks, moved to 36 GB
disks from a variety of manufacturers, and then to 72 GB IBM disks
before this last capacity upgrade).
Sun branded Qlogic 2310 FC cards (375-3102). Sun qlc drivers and MPIO
is enabled.
The rest of the system:
2 CPU Opteron board and chips(>2GHZ), 8GB RAM.
When a hard drive fails in the enclosure, it bypasses the bad drive
and turns on a light to let me know a disk failure has happened. This
never happens with this event, pointing it to be a software problem.
Once it goes off the rails and starts off-lining disks it causes the
system to have problems. Login for a user takes forever (40 minutes
minimum to pass the last login message), any command touching on
storage or zfs/zpool hangs for just as long.
I can reliably reproduce the issue by either copying a large amount of
data into the pool or running a scrub.
All disks test fine via destructive tests in format.
I just reproduced it by clearing and creating anew pool called share:
# zpool status share
pool: share
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
share ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c0t50050767190B6C76d0 ONLINE 0 0 0
c0t500507671908E72Bd0 ONLINE 0 0 0
c0t500507671907A32Ad0 ONLINE 0 0 0
c0t50050767190C4CFDd0 ONLINE 0 0 0
c0t500507671906704Dd0 ONLINE 0 0 0
c0t500507671918892Ad0 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c0t50050767190D11E4d0 ONLINE 0 0 0
c0t500507671915CABEd0 ONLINE 0 0 0
c0t50050767191371C7d0 ONLINE 0 0 0
c0t5005076719125EDBd0 ONLINE 0 0 0
c0t50050767190E4DABd0 ONLINE 0 0 0
c0t5005076719147ECAd0 ONLINE 0 0 0
errors: No known data errors
messages logs something like the following:
May 21 15:27:54 solarisfc scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):
May 21 15:27:54 solarisfc /scsi_vhci/disk at g50050767191371c7
(sd2): Command Timeout on path
/pci at 0,0/pci1022,7450 at a/pci1077,106 at 3/fp at 0,0 (fp1)
May 21 15:27:54 solarisfc scsi: [ID 107833 kern.warning] WARNING:
/scsi_vhci/disk at g50050767191371c7 (sd2):
May 21 15:27:54 solarisfc SCSI transport failed: reason
''timeout'': retrying command
May 21 15:27:54 solarisfc scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):
May 21 15:27:54 solarisfc /scsi_vhci/disk at g50050767191371c7
(sd2): Command Timeout on path
/pci at 0,0/pci1022,7450 at a/pci1077,106 at 2/fp at 0,0 (fp0)
May 21 15:28:54 solarisfc scsi: [ID 107833 kern.warning] WARNING:
/scsi_vhci/disk at g50050767191371c7 (sd2):
May 21 15:28:54 solarisfc SCSI transport failed: reason
''timeout'': giving up
May 21 15:32:54 solarisfc scsi: [ID 107833 kern.warning] WARNING:
/scsi_vhci/disk at g50050767191371c7 (sd2):
May 21 15:32:54 solarisfc SYNCHRONIZE CACHE command failed (5)
May 21 15:40:54 solarisfc scsi: [ID 107833 kern.warning] WARNING:
/scsi_vhci/disk at g50050767191371c7 (sd2):
May 21 15:40:54 solarisfc drive offline
May 21 15:48:55 solarisfc scsi: [ID 107833 kern.warning] WARNING:
/scsi_vhci/disk at g50050767191371c7 (sd2):
May 21 15:48:55 solarisfc drive offline
May 21 15:56:55 solarisfc scsi: [ID 107833 kern.warning] WARNING:
/scsi_vhci/disk at g50050767191371c7 (sd2):
May 21 15:56:55 solarisfc drive offline
May 21 16:04:55 solarisfc scsi: [ID 107833 kern.warning] WARNING:
/scsi_vhci/disk at g50050767191371c7 (sd2):
May 21 16:04:55 solarisfc drive offline
May 21 16:04:56 solarisfc fmd: [ID 441519 daemon.error] SUNW-MSG-ID:
ZFS-8000-FD, TYPE: Fault, VER: 1, SEVERITY: Major
May 21 16:04:56 solarisfc EVENT-TIME: Fri May 21 16:04:56 EDT 2010
May 21 16:04:56 solarisfc PLATFORM: To Be Filled By O.E.M., CSN: To Be
Filled By O.E.M., HOSTNAME: solarisfc
May 21 16:04:56 solarisfc SOURCE: zfs-diagnosis, REV: 1.0
May 21 16:04:56 solarisfc EVENT-ID: 295d7729-9a93-47f1-de9d-ba3a08b2d477
May 21 16:04:56 solarisfc DESC: The number of I/O errors associated
with a ZFS device exceeded
May 21 16:04:56 solarisfc acceptable levels. Refer to
http://sun.com/msg/ZFS-8000-FD for more information.
May 21 16:04:56 solarisfc AUTO-RESPONSE: The device has been offlined
and marked as faulted. An attempt
May 21 16:04:56 solarisfc will be made to activate a hot
spare if available.
May 21 16:04:56 solarisfc IMPACT: Fault tolerance of the pool may be
compromised.
May 21 16:04:56 solarisfc REC-ACTION: Run ''zpool status -x''
and
replace the bad device.
Fmdump reports only ZFS errors:
# fmdump -e
TIME CLASS
May 21 15:28:54.1367 ereport.fs.zfs.io
May 21 15:28:54.1367 ereport.fs.zfs.io
May 21 15:28:54.1367 ereport.fs.zfs.io
May 21 15:28:54.1369 ereport.fs.zfs.probe_failure
May 21 16:04:55.8976 ereport.fs.zfs.vdev.open_failed
The pool looks like this:
# zpool status share
pool: share
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use ''zpool clear'' to
mark the device
repaired.
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
share DEGRADED 0 0 0
raidz2 ONLINE 0 0 0
c0t50050767190B6C76d0 ONLINE 0 0 0
c0t500507671908E72Bd0 ONLINE 0 0 0
c0t500507671907A32Ad0 ONLINE 0 0 0
c0t50050767190C4CFDd0 ONLINE 0 0 0
c0t500507671906704Dd0 ONLINE 0 0 0
c0t500507671918892Ad0 ONLINE 0 0 0
raidz2 DEGRADED 0 0 0
c0t50050767190D11E4d0 ONLINE 0 0 0
c0t500507671915CABEd0 ONLINE 0 0 0
c0t50050767191371C7d0 FAULTED 3 50 0 too many errors
c0t5005076719125EDBd0 ONLINE 0 0 0
c0t50050767190E4DABd0 ONLINE 0 0 0
c0t5005076719147ECAd0 ONLINE 0 0 0
If I had not broken out of the large data copy (700GB of 340GB
completed) it would have kept going and off-lined more disks.
If I had any hot spares in the pool (I have 2 disks for that but do
not have them in the pool right now) it would have tried to put them
in and the last time that happened it put one in overloaded the I/O
and failed it then did the same with my second hot spare.
fcinfo reports the following for each controller:
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 1
Loss of Signal Count: 1
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
Here is the detailed FC information:
# fcinfo remote-port -slp 210000e08b80c45c
Remote Port WWN: 50050767198e4dab
Active FC4 Types:
SCSI Target: yes
Node WWN: 50050767190e4dab
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 11
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 1617
Invalid CRC Count: 0
LUN: 0
Vendor: IBM
Product: IC35L146F2DY10-0
OS Device Name: /dev/rdsk/c0t50050767190E4DABd0s2
Remote Port WWN: 50050767199371c7
Active FC4 Types:
SCSI Target: yes
Node WWN: 50050767191371c7
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 11
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 1112
Invalid CRC Count: 0
LUN: 0
Vendor: IBM
Product: IC35L146F2DY10-0
OS Device Name: /dev/rdsk/c0t50050767191371C7d0s2
Remote Port WWN: 500507671998892a
Active FC4 Types:
SCSI Target: yes
Node WWN: 500507671918892a
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 1
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 155
Invalid CRC Count: 0
LUN: 0
Vendor: IBM
Product: IC35L146F2DY10-0
OS Device Name: /dev/rdsk/c0t500507671918892Ad0s2
Remote Port WWN: 5005076719925edb
Active FC4 Types:
SCSI Target: yes
Node WWN: 5005076719125edb
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 1677
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 6735
Invalid CRC Count: 0
LUN: 0
Vendor: IBM
Product: IC35L146F2DY10-0
OS Device Name: /dev/rdsk/c0t5005076719125EDBd0s2
Remote Port WWN: 50050767198b6c76
Active FC4 Types:
SCSI Target: yes
Node WWN: 50050767190b6c76
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 1
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 161
Invalid CRC Count: 0
LUN: 0
Vendor: IBM
Product: IC35L146F2DY10-0
OS Device Name: /dev/rdsk/c0t50050767190B6C76d0s2
Remote Port WWN: 50050767198d60c6
Active FC4 Types:
SCSI Target: yes
Node WWN: 50050767190d60c6
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 1
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 424
Invalid CRC Count: 0
LUN: 0
Vendor: IBM
Product: IC35L146F2DY10-0
OS Device Name: /dev/rdsk/c0t50050767190D60C6d0s2
Remote Port WWN: 500507671987a32a
Active FC4 Types:
SCSI Target: yes
Node WWN: 500507671907a32a
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 1
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 157
Invalid CRC Count: 0
LUN: 0
Vendor: IBM
Product: IC35L146F2DY10-0
OS Device Name: /dev/rdsk/c0t500507671907A32Ad0s2
Remote Port WWN: 50050767198c4cfd
Active FC4 Types:
SCSI Target: yes
Node WWN: 50050767190c4cfd
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 1
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 157
Invalid CRC Count: 0
LUN: 0
Vendor: IBM
Product: IC35L146F2DY10-0
OS Device Name: /dev/rdsk/c0t50050767190C4CFDd0s2
Remote Port WWN: 508002000000d073
Active FC4 Types:
SCSI Target: unknown
Node WWN: 508002000000d070
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
Error has occured. HBA_ScsiReportLUNsV2 failed. reason SCSI CHECK CONDITION
Remote Port WWN: 508002000000d074
Active FC4 Types:
SCSI Target: unknown
Node WWN: 508002000000d070
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
Error has occured. HBA_ScsiReportLUNsV2 failed. reason SCSI CHECK CONDITION
Remote Port WWN: 500507671988e72b
Active FC4 Types:
SCSI Target: yes
Node WWN: 500507671908e72b
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 1
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 154
Invalid CRC Count: 0
LUN: 0
Vendor: IBM
Product: IC35L146F2DY10-0
OS Device Name: /dev/rdsk/c0t500507671908E72Bd0s2
Remote Port WWN: 5005076719947eca
Active FC4 Types:
SCSI Target: yes
Node WWN: 5005076719147eca
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 11
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 853
Invalid CRC Count: 0
LUN: 0
Vendor: IBM
Product: IC35L146F2DY10-0
OS Device Name: /dev/rdsk/c0t5005076719147ECAd0s2
Remote Port WWN: 500507671995cabe
Active FC4 Types:
SCSI Target: yes
Node WWN: 500507671915cabe
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 71
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 464
Invalid CRC Count: 0
LUN: 0
Vendor: IBM
Product: IC35L146F2DY10-0
OS Device Name: /dev/rdsk/c0t500507671915CABEd0s2
Remote Port WWN: 500507671997f832
Active FC4 Types:
SCSI Target: yes
Node WWN: 500507671917f832
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 14
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 837
Invalid CRC Count: 0
LUN: 0
Vendor: IBM
Product: IC35L146F2DY10-0
OS Device Name: /dev/rdsk/c0t500507671917F832d0s2
Remote Port WWN: 50050767198d11e4
Active FC4 Types:
SCSI Target: yes
Node WWN: 50050767190d11e4
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 20
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 862
Invalid CRC Count: 0
LUN: 0
Vendor: IBM
Product: IC35L146F2DY10-0
OS Device Name: /dev/rdsk/c0t50050767190D11E4d0s2
Remote Port WWN: 500507671986704d
Active FC4 Types:
SCSI Target: yes
Node WWN: 500507671906704d
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 1
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 157
Invalid CRC Count: 0
LUN: 0
Vendor: IBM
Product: IC35L146F2DY10-0
OS Device Name: /dev/rdsk/c0t500507671906704Dd0s2
Unless there is a way to tell ZFS or the FC bus to chill out and not
overload things I don''t see how I can make use of my hardware in
Solaris 10.
I''ve seen this happening on other Solaris 10 (x86 only) boxes in a
data center environment rather then my basement as well.
It''s starting to look like I need to move the FC stuff to AIX, Linux,
Windows or Solaris10 circa 2006 for reliability, which is sad because
then I will no longer be able to use ZFS which has served me well for
years.
Bob Friesenhahn
2010-May-22 15:33 UTC
[zfs-discuss] ZFS no longer working with FC devices.
On Fri, 21 May 2010, Demian Phillips wrote:> For years I have been running a zpool using a Fibre Channel array with > no problems. I would scrub every so often and dump huge amounts of > data (tens or hundreds of GB) around and it never had a problem > outside of one confirmed (by the array) disk failure. > > I upgraded to sol10x86 05/09 last year and since then I have > discovered any sufficiently high I/O from ZFS starts causing timeouts > and off-lining disks. This leads to failure (once rebooted and cleaned > all is well) long term because you can no longer scrub reliably.The problem could be with the device driver, your FC card, or the array itself. In my case, issues I thought were to blame on my motherboard or Solaris were due to a defective FC card and replacing the card resolved the problem. If the problem is that your storage array is becoming overloaded with requests, then try adding this to your /etc/system file: * Set device I/O maximum concurrency * http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Device_I.2FO_Queue_Size_.28I.2FO_Concurrency.29 set zfs:zfs_vdev_max_pending = 5 Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Demian Phillips
2010-May-23 13:01 UTC
[zfs-discuss] ZFS no longer working with FC devices.
On Sat, May 22, 2010 at 11:33 AM, Bob Friesenhahn <bfriesen at simple.dallas.tx.us> wrote:> On Fri, 21 May 2010, Demian Phillips wrote: > >> For years I have been running a zpool using a Fibre Channel array with >> no problems. I would scrub every so often and dump huge amounts of >> data (tens or hundreds of GB) around and it never had a problem >> outside of one confirmed (by the array) disk failure. >> >> I upgraded to sol10x86 05/09 last year and since then I have >> discovered any sufficiently high I/O from ZFS starts causing timeouts >> and off-lining disks. This leads to failure (once rebooted and cleaned >> all is well) long term because you can no longer scrub reliably. > > The problem could be with the device driver, your FC card, or the array > itself. ?In my case, issues I thought were to blame on my motherboard or > Solaris were due to a defective FC card and replacing the card resolved the > problem. > > If the problem is that your storage array is becoming overloaded with > requests, then try adding this to your /etc/system file: > > * Set device I/O maximum concurrency > * > http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Device_I.2FO_Queue_Size_.28I.2FO_Concurrency.29 > set zfs:zfs_vdev_max_pending = 5 > > Bob > -- > Bob Friesenhahn > bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer, ? ?http://www.GraphicsMagick.org/ >I''ve gone back to Solaris 10 11/06. It''s working fine, but I notice some differences in performance that are I think key to the problem. With the latest Solaris 10 (u8) throughput according to zpool iostat was hitting about 115MB/sec sometimes a little higher. With 11/06 it maxes out at 40MB/sec. Both setups are using mpio devices as far as I can tell. Next is to go back to u8 and see if the tuning you suggested will help. It really looks to me that the OS is asking too much of the FC chain I have. The really puzzling thing is I just got told about a brand new Dell Solaris x86 production box using current and supported FC devices and a supported SAN get the same kind of problems when a scrub is run. I''m going to investigate that and see if we can get a fix from Oracle as that does have a support contract. It may shed some light on the issue I am seeing on the older hardware.
On May 23, 2010, at 6:01 AM, Demian Phillips wrote:> On Sat, May 22, 2010 at 11:33 AM, Bob Friesenhahn > <bfriesen at simple.dallas.tx.us> wrote: >> On Fri, 21 May 2010, Demian Phillips wrote: >> >>> For years I have been running a zpool using a Fibre Channel array with >>> no problems. I would scrub every so often and dump huge amounts of >>> data (tens or hundreds of GB) around and it never had a problem >>> outside of one confirmed (by the array) disk failure. >>> >>> I upgraded to sol10x86 05/09 last year and since then I have >>> discovered any sufficiently high I/O from ZFS starts causing timeouts >>> and off-lining disks. This leads to failure (once rebooted and cleaned >>> all is well) long term because you can no longer scrub reliably. >> >> The problem could be with the device driver, your FC card, or the array >> itself. In my case, issues I thought were to blame on my motherboard or >> Solaris were due to a defective FC card and replacing the card resolved the >> problem. >> >> If the problem is that your storage array is becoming overloaded with >> requests, then try adding this to your /etc/system file: >> >> * Set device I/O maximum concurrency >> * >> http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Device_I.2FO_Queue_Size_.28I.2FO_Concurrency.29 >> set zfs:zfs_vdev_max_pending = 5I would lower it even farther. Perhaps 2.> I''ve gone back to Solaris 10 11/06. > It''s working fine, but I notice some differences in performance that > are I think key to the problem.Yep, lots of performance improvements were added later.> With the latest Solaris 10 (u8) throughput according to zpool iostat > was hitting about 115MB/sec sometimes a little higher.That should be about right for the A5100.> With 11/06 it maxes out at 40MB/sec. > > Both setups are using mpio devices as far as I can tell. > > Next is to go back to u8 and see if the tuning you suggested will > help. It really looks to me that the OS is asking too much of the FC > chain I have.I think that is a nice way of saying it.> The really puzzling thing is I just got told about a brand new Dell > Solaris x86 production box using current and supported FC devices and > a supported SAN get the same kind of problems when a scrub is run. I''m > going to investigate that and see if we can get a fix from Oracle as > that does have a support contract. It may shed some light on the issue > I am seeing on the older hardware.The scrub workload is no different than any other stress test. I''m sure you can run a benchmark or three on the raw device and get the same error messages. FWIW, the A5100 went end-of-life (EOL) in 2001 and end-of-service-life (EOSL) in 2006. Personally, I hate them with a passion and would like to extend an offer to use my tractor to bury the beast :-). -- richard -- ZFS and NexentaStor training, Rotterdam, July 13-15, 2010 http://nexenta-rotterdam.eventbrite.com/
On 5/23/2010 11:49 AM, Richard Elling wrote:> FWIW, the A5100 went end-of-life (EOL) in 2001 and end-of-service-life > (EOSL) in 2006. Personally, I hate them with a passion and would like to > extend an offer to use my tractor to bury the beast:-).I''m sure I can get some others to help. Can I smash the gbics? Those were my favorite. :-)
Demian Phillips
2010-May-24 11:06 UTC
[zfs-discuss] ZFS no longer working with FC devices.
On Sun, May 23, 2010 at 12:02 PM, Torrey McMahon <tmcmahon2 at yahoo.com> wrote:> ?On 5/23/2010 11:49 AM, Richard Elling wrote: >> >> FWIW, the A5100 went end-of-life (EOL) in 2001 and end-of-service-life >> (EOSL) in 2006. Personally, I ?hate them with a passion and would like to >> extend an offer to use my tractor to bury the beast:-). > > I''m sure I can get some others to help. Can I smash the gbics? Those were my > favorite. :-) > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >I''d be more then happy to take someone up on the offer but I''d need a good deal on more current FC array. Since this is my home environment I am limited by my insignificant pay and the wife factor (who does indulge me from time to time). Without a corporate IT budget I make do with everything from free to what I can afford used. To be honest I''d rather be using an IBM DS4K series array. Current stress test is creating 700 (50% of array capacity) 1GB files from /dev/urandom and then I will scrub. If all goes well it''s back to u8 and tuning it.
On May 24, 2010, at 4:06 AM, Demian Phillips wrote:> On Sun, May 23, 2010 at 12:02 PM, Torrey McMahon <tmcmahon2 at yahoo.com> wrote: >> On 5/23/2010 11:49 AM, Richard Elling wrote: >>> >>> FWIW, the A5100 went end-of-life (EOL) in 2001 and end-of-service-life >>> (EOSL) in 2006. Personally, I hate them with a passion and would like to >>> extend an offer to use my tractor to bury the beast:-). >> >> I''m sure I can get some others to help. Can I smash the gbics? Those were my >> favorite. :-) >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> > > I''d be more then happy to take someone up on the offer but I''d need a > good deal on more current FC array. Since this is my home environment > I am limited by my insignificant pay and the wife factor (who does > indulge me from time to time). Without a corporate IT budget I make do > with everything from free to what I can afford used. > > To be honest I''d rather be using an IBM DS4K series array. > > Current stress test is creating 700 (50% of array capacity) 1GB files > from /dev/urandom and then I will scrub.Unfortunately, /dev/urandom is too slow for direct stress testing. It can be used as a seed for random data files that are then used for stress testing. -- richard -- ZFS and NexentaStor training, Rotterdam, July 13-15, 2010 http://nexenta-rotterdam.eventbrite.com/
Andrew Daugherity
2010-May-24 21:06 UTC
[zfs-discuss] ZFS no longer working with FC devices.
I had a similar problem with a RAID shelf (switched to JBOD mode, with each physical disk presented as a LUN) connected via FC (qlc driver, but no MPIO). Running a scrub would eventually generate I/O errors and many messages like this: Sep 6 15:12:53 imsfs scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci10de,5d at e/pci1077,142 at 0/fp at 0,0/disk at w2100 0004d960cdec,e (sd4): Sep 6 15:12:53 imsfs Request Sense couldn''t get sense data and eventually one or more disks would get marked as faulted by ZFS. This was under s10u6 (10/08, I think) but I imagine it still holds for u8. I did not have these problems with just one or two LUNs presented from the array, but I prefer to run ZFS in the recommended configuration where it manages the disks. My storage vendor (3rd-party, not Sun) recommended that in /etc/system I add ''set ssd:ssd_max_throttle = 23'' or less and ''set ssd:ssd_io_time = 0x60'' or 0x78. The default 0x20 (in what version of Solaris?) is apparently not enough in many cases. In my case (x64) I discovered I needed sd:sd_max_throttle, etc. (not ssd, which is apparently only for sparc), and that the default sd_io_time on recent Solaris 10 already is 0x60. Apparently the general rule for max_throttle is 256/# of LUNs, but my vendor found that 23 was the maximum reliable setting for 16 LUNs. This may or may not help you but it''s something to try. Without the max_throttle setting, I would get errors somewhere between 30 minutes and 4 hours into a scrub, and with it scrubs run successfully. -Andrew>>> Demian Phillips <demianphillips at gmail.com> 5/23/2010 8:01 AM >>>On Sat, May 22, 2010 at 11:33 AM, Bob Friesenhahn <bfriesen at simple.dallas.tx.us> wrote:> On Fri, 21 May 2010, Demian Phillips wrote: > >> For years I have been running a zpool using a Fibre Channel array with >> no problems. I would scrub every so often and dump huge amounts of >> data (tens or hundreds of GB) around and it never had a problem >> outside of one confirmed (by the array) disk failure. >> >> I upgraded to sol10x86 05/09 last year and since then I have >> discovered any sufficiently high I/O from ZFS starts causing timeouts >> and off-lining disks. This leads to failure (once rebooted and cleaned >> all is well) long term because you can no longer scrub reliably. > > The problem could be with the device driver, your FC card, or the array > itself. In my case, issues I thought were to blame on my motherboard or > Solaris were due to a defective FC card and replacing the card resolved the > problem. > > If the problem is that your storage array is becoming overloaded with > requests, then try adding this to your /etc/system file: > > * Set device I/O maximum concurrency > * > http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Device_I.2FO_Queue_Size_.28I.2FO_Concurrency.29 > set zfs:zfs_vdev_max_pending = 5 > > Bob > -- > Bob Friesenhahn > bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ >I''ve gone back to Solaris 10 11/06. It''s working fine, but I notice some differences in performance that are I think key to the problem. With the latest Solaris 10 (u8) throughput according to zpool iostat was hitting about 115MB/sec sometimes a little higher. With 11/06 it maxes out at 40MB/sec. Both setups are using mpio devices as far as I can tell. Next is to go back to u8 and see if the tuning you suggested will help. It really looks to me that the OS is asking too much of the FC chain I have. The really puzzling thing is I just got told about a brand new Dell Solaris x86 production box using current and supported FC devices and a supported SAN get the same kind of problems when a scrub is run. I''m going to investigate that and see if we can get a fix from Oracle as that does have a support contract. It may shed some light on the issue I am seeing on the older hardware.