Morten-Christian Bernson
2009-Dec-28 00:52 UTC
[zfs-discuss] Extremely bad performance - hw failure?
Lately my zfs pool in my home server has degraded to a state where it can be
said it doesn''t work at all. Read spead is slower than I can read from
the internet on my slow dsl-line... This is compared to just a short while ago,
where I could read from it with over 50mb/sec over the network.
My setup:
Running latest Solaris 10: # uname -a
SunOS solssd01 5.10 Generic_142901-02 i86pc i386 i86pc
# zpool status DATA
pool: DATA
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
DATA ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c2t5d0 ONLINE 0 0 0
c2t4d0 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0
c2t2d0 ONLINE 0 0 0
spares
c0t2d0 AVAIL
errors: No known data errors
# zfs list -r DATA
NAME USED AVAIL REFER MOUNTPOINT
DATA 3,78T 229G 3,78T /DATA
All of the drives in this pool are 1.5tb western digital green drives. I am not
seeing any error messages in /var/adm/messages, and "fmdump -eV" shows
no errors... However, I am seeing some soft faults in "iostat -eEn":
---- errors ---
s/w h/w trn tot device
2 0 0 2 c0t0d0
1 0 0 1 c1t0d0
2 0 0 2 c2t1d0
151 0 0 151 c2t2d0
151 0 0 151 c2t3d0
153 0 0 153 c2t4d0
153 0 0 153 c2t5d0
2 0 0 2 c0t1d0
3 0 0 3 c0t2d0
0 0 0 0 solssd01:vold(pid531)
c0t0d0 Soft Errors: 2 Hard Errors: 0 Transport Errors: 0
Vendor: Sun Product: STK RAID INT Revision: V1.0 Serial No:
Size: 31.87GB <31866224128 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 2 Predictive Failure Analysis: 0
c1t0d0 Soft Errors: 1 Hard Errors: 0 Transport Errors: 0
Vendor: _NEC Product: DVD_RW ND-3500AG Revision: 2.16 Serial No:
Size: 0.00GB <0 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
c2t1d0 Soft Errors: 2 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: SAMSUNG HD753LJ Revision: 1113 Serial No:
Size: 750.16GB <750156373504 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 2 Predictive Failure Analysis: 0
c2t2d0 Soft Errors: 151 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: WDC WD15EADS-00R Revision: 0A01 Serial No:
Size: 1500.30GB <1500301909504 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 151 Predictive Failure Analysis: 0
c2t3d0 Soft Errors: 151 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: WDC WD15EADS-00R Revision: 0A01 Serial No:
Size: 1500.30GB <1500301909504 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 151 Predictive Failure Analysis: 0
c2t4d0 Soft Errors: 153 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: WDC WD15EADS-00R Revision: 0A01 Serial No:
Size: 1500.30GB <1500301909504 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 153 Predictive Failure Analysis: 0
c2t5d0 Soft Errors: 153 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: WDC WD15EADS-00R Revision: 0A01 Serial No:
Size: 1500.30GB <1500301909504 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 153 Predictive Failure Analysis: 0
c0t1d0 Soft Errors: 2 Hard Errors: 0 Transport Errors: 0
Vendor: Sun Product: STK RAID INT Revision: V1.0 Serial No:
Size: 31.87GB <31866224128 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 2 Predictive Failure Analysis: 0
c0t2d0 Soft Errors: 3 Hard Errors: 0 Transport Errors: 0
Vendor: Sun Product: STK RAID INT Revision: V1.0 Serial No:
Size: 1497.86GB <1497859358208 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 3 Predictive Failure Analysis: 0
I am curious as to why the counter for "Illegal request" goes up all
the time. The machine was rebooted ~11 hours ago, and it goes up all the time
when I try to use the pool...
The machine is a quite powerful one, and top shows no cpu load, no iowait and
plenty of available memory. The machine basicly doesn''t do anything at
the moment, still it can take several minutes to copy a 300mb file from
somewhere in the pool to /tmp/...
# top
last pid: 1383; load avg: 0.01, 0.00, 0.00; up 0+10:47:57
01:39:17
55 processes: 54 sleeping, 1 on cpu
CPU states: 99.0% idle, 0.0% user, 1.0% kernel, 0.0% iowait, 0.0% swap
Kernel: 193 ctxsw, 3 trap, 439 intr, 298 syscall, 3 flt
Memory: 8186M phys mem, 4699M free mem, 2048M total swap, 2048M free swap
I thought I might have run into problems described here on the forums with the
ARC and fragmentation, but it doesn''t seem so:
# echo "::arc"|mdb -k
hits = 490044
misses = 37004
demand_data_hits = 282392
demand_data_misses = 2113
demand_metadata_hits = 191757
demand_metadata_misses = 21034
prefetch_data_hits = 851
prefetch_data_misses = 10265
prefetch_metadata_hits = 15044
prefetch_metadata_misses = 3592
mru_hits = 73416
mru_ghost_hits = 16
mfu_hits = 401500
mfu_ghost_hits = 24
deleted = 1555
recycle_miss = 0
mutex_miss = 0
evict_skip = 1487
hash_elements = 37032
hash_elements_max = 37045
hash_collisions = 10094
hash_chains = 4365
hash_chain_max = 4
p = 3576 MB
c = 7154 MB
c_min = 894 MB
c_max = 7154 MB
size = 1797 MB
hdr_size = 8002680
data_size = 1866272256
other_size = 10519712
l2_hits = 0
l2_misses = 0
l2_feeds = 0
l2_rw_clash = 0
l2_read_bytes = 0
l2_write_bytes = 0
l2_writes_sent = 0
l2_writes_done = 0
l2_writes_error = 0
l2_writes_hdr_miss = 0
l2_evict_lock_retry = 0
l2_evict_reading = 0
l2_free_on_write = 0
l2_abort_lowmem = 0
l2_cksum_bad = 0
l2_io_error = 0
l2_size = 0
l2_hdr_size = 0
memory_throttle_count = 0
arc_no_grow = 0
arc_tempreserve = 0 MB
arc_meta_used = 372 MB
arc_meta_limit = 1788 MB
arc_meta_max = 372 MB
I then tried to start a scrub, and it seems like it will take forever... It used
to take a few hours, now it says it will be done in almost 700 hours:
scrub: scrub in progress for 4h43m, 0,68% done, 685h2m to go
Does anyone have any clue as to what is happening, and what I can do? If a disk
is failing without the OS noticing, it would be nice to find a way to know which
drive it is, and get it exchanged before it''s too late.
All help is apreciated...
Yours sincerly,
Morten-Christian Bernson
--
This message posted from opensolaris.org
Richard Elling
2009-Dec-28 01:27 UTC
[zfs-discuss] Extremely bad performance - hw failure?
The best place to start looking at disk-related performance problems is iostat. Slow disks will show high service times. There are many options, but I usually use something like: iostat -zxcnPT d 1 Ignore the first line. Look at the service times. They should be below 10ms for good performance. -- richard On Dec 27, 2009, at 4:52 PM, Morten-Christian Bernson wrote:> Lately my zfs pool in my home server has degraded to a state where > it can be said it doesn''t work at all. Read spead is slower than I > can read from the internet on my slow dsl-line... This is compared > to just a short while ago, where I could read from it with over 50mb/ > sec over the network. > > My setup: > Running latest Solaris 10: # uname -a > SunOS solssd01 5.10 Generic_142901-02 i86pc i386 i86pc > > # zpool status DATA > pool: DATA > state: ONLINE > config: > NAME STATE READ WRITE CKSUM > DATA ONLINE 0 0 0 > raidz1 ONLINE 0 0 0 > c2t5d0 ONLINE 0 0 0 > c2t4d0 ONLINE 0 0 0 > c2t3d0 ONLINE 0 0 0 > c2t2d0 ONLINE 0 0 0 > spares > c0t2d0 AVAIL > errors: No known data errors > > # zfs list -r DATA > NAME USED AVAIL REFER MOUNTPOINT > DATA 3,78T 229G 3,78T /DATA > > All of the drives in this pool are 1.5tb western digital green > drives. I am not seeing any error messages in /var/adm/messages, and > "fmdump -eV" shows no errors... However, I am seeing some soft > faults in "iostat -eEn": > ---- errors --- > s/w h/w trn tot device > 2 0 0 2 c0t0d0 > 1 0 0 1 c1t0d0 > 2 0 0 2 c2t1d0 > 151 0 0 151 c2t2d0 > 151 0 0 151 c2t3d0 > 153 0 0 153 c2t4d0 > 153 0 0 153 c2t5d0 > 2 0 0 2 c0t1d0 > 3 0 0 3 c0t2d0 > 0 0 0 0 solssd01:vold(pid531) > c0t0d0 Soft Errors: 2 Hard Errors: 0 Transport Errors: 0 > Vendor: Sun Product: STK RAID INT Revision: V1.0 Serial No: > Size: 31.87GB <31866224128 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > Illegal Request: 2 Predictive Failure Analysis: 0 > c1t0d0 Soft Errors: 1 Hard Errors: 0 Transport Errors: 0 > Vendor: _NEC Product: DVD_RW ND-3500AG Revision: 2.16 Serial No: > Size: 0.00GB <0 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > Illegal Request: 1 Predictive Failure Analysis: 0 > c2t1d0 Soft Errors: 2 Hard Errors: 0 Transport Errors: 0 > Vendor: ATA Product: SAMSUNG HD753LJ Revision: 1113 Serial No: > Size: 750.16GB <750156373504 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > Illegal Request: 2 Predictive Failure Analysis: 0 > c2t2d0 Soft Errors: 151 Hard Errors: 0 Transport Errors: 0 > Vendor: ATA Product: WDC WD15EADS-00R Revision: 0A01 Serial No: > Size: 1500.30GB <1500301909504 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > Illegal Request: 151 Predictive Failure Analysis: 0 > c2t3d0 Soft Errors: 151 Hard Errors: 0 Transport Errors: 0 > Vendor: ATA Product: WDC WD15EADS-00R Revision: 0A01 Serial No: > Size: 1500.30GB <1500301909504 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > Illegal Request: 151 Predictive Failure Analysis: 0 > c2t4d0 Soft Errors: 153 Hard Errors: 0 Transport Errors: 0 > Vendor: ATA Product: WDC WD15EADS-00R Revision: 0A01 Serial No: > Size: 1500.30GB <1500301909504 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > Illegal Request: 153 Predictive Failure Analysis: 0 > c2t5d0 Soft Errors: 153 Hard Errors: 0 Transport Errors: 0 > Vendor: ATA Product: WDC WD15EADS-00R Revision: 0A01 Serial No: > Size: 1500.30GB <1500301909504 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > Illegal Request: 153 Predictive Failure Analysis: 0 > c0t1d0 Soft Errors: 2 Hard Errors: 0 Transport Errors: 0 > Vendor: Sun Product: STK RAID INT Revision: V1.0 Serial No: > Size: 31.87GB <31866224128 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > Illegal Request: 2 Predictive Failure Analysis: 0 > c0t2d0 Soft Errors: 3 Hard Errors: 0 Transport Errors: 0 > Vendor: Sun Product: STK RAID INT Revision: V1.0 Serial No: > Size: 1497.86GB <1497859358208 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > Illegal Request: 3 Predictive Failure Analysis: 0 > > I am curious as to why the counter for "Illegal request" goes up all > the time. The machine was rebooted ~11 hours ago, and it goes up > all the time when I try to use the pool... > > The machine is a quite powerful one, and top shows no cpu load, no > iowait and plenty of available memory. The machine basicly doesn''t > do anything at the moment, still it can take several minutes to copy > a 300mb file from somewhere in the pool to /tmp/... > # top > last pid: 1383; load avg: 0.01, 0.00, 0.00; up > 0 > + > 10 > : > 47 > :57 > 01:39:17 > 55 processes: 54 sleeping, 1 on cpu > CPU states: 99.0% idle, 0.0% user, 1.0% kernel, 0.0% iowait, > 0.0% swap > Kernel: 193 ctxsw, 3 trap, 439 intr, 298 syscall, 3 flt > Memory: 8186M phys mem, 4699M free mem, 2048M total swap, 2048M free > swap > > I thought I might have run into problems described here on the > forums with the ARC and fragmentation, but it doesn''t seem so: > # echo "::arc"|mdb -k > hits = 490044 > misses = 37004 > demand_data_hits = 282392 > demand_data_misses = 2113 > demand_metadata_hits = 191757 > demand_metadata_misses = 21034 > prefetch_data_hits = 851 > prefetch_data_misses = 10265 > prefetch_metadata_hits = 15044 > prefetch_metadata_misses = 3592 > mru_hits = 73416 > mru_ghost_hits = 16 > mfu_hits = 401500 > mfu_ghost_hits = 24 > deleted = 1555 > recycle_miss = 0 > mutex_miss = 0 > evict_skip = 1487 > hash_elements = 37032 > hash_elements_max = 37045 > hash_collisions = 10094 > hash_chains = 4365 > hash_chain_max = 4 > p = 3576 MB > c = 7154 MB > c_min = 894 MB > c_max = 7154 MB > size = 1797 MB > hdr_size = 8002680 > data_size = 1866272256 > other_size = 10519712 > l2_hits = 0 > l2_misses = 0 > l2_feeds = 0 > l2_rw_clash = 0 > l2_read_bytes = 0 > l2_write_bytes = 0 > l2_writes_sent = 0 > l2_writes_done = 0 > l2_writes_error = 0 > l2_writes_hdr_miss = 0 > l2_evict_lock_retry = 0 > l2_evict_reading = 0 > l2_free_on_write = 0 > l2_abort_lowmem = 0 > l2_cksum_bad = 0 > l2_io_error = 0 > l2_size = 0 > l2_hdr_size = 0 > memory_throttle_count = 0 > arc_no_grow = 0 > arc_tempreserve = 0 MB > arc_meta_used = 372 MB > arc_meta_limit = 1788 MB > arc_meta_max = 372 MB > > I then tried to start a scrub, and it seems like it will take > forever... It used to take a few hours, now it says it will be done > in almost 700 hours: > scrub: scrub in progress for 4h43m, 0,68% done, 685h2m to go > > Does anyone have any clue as to what is happening, and what I can > do? If a disk is failing without the OS noticing, it would be nice > to find a way to know which drive it is, and get it exchanged before > it''s too late. > > All help is apreciated... > > Yours sincerly, > Morten-Christian Bernson > -- > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
I''ve had this happen to me too. I found some dtrace scripts at the time that showed that the file system was spending too much time finding available 128k blocks or the like as I was near full per each disk, even though combined I still had 140GB left of my 3TB pool. The SPA code I believe it was was spending too much time walking the available pool for continguous space for new writes, and this affecting both read and write performance dramatically (measured in kb/sec). I was able to alleviate the pressure so to speak by adjusting the recordsize for the pool down to 8k (32k is likely more recommended) and from there I could then start to clear out space. Anything below 10% available space seems to cause ZFS to start behaving poorly, and getting down lower increases the problems. But the root cause was metadata management on pools w/ less than 5-10% disk space left. In my case, I had lots of symlinks, lots of small files, and also dozens of snapshots. My pool was a RAID10 (aka, 3 mirror sets striped). On Sun, Dec 27, 2009 at 4:52 PM, Morten-Christian Bernson <mcb at uib.no> wrote:> Lately my zfs pool in my home server has degraded to a state where it can be said it doesn''t work at all. ?Read spead is slower than I can read from the internet on my slow dsl-line... This is compared to just a short while ago, where I could read from it with over 50mb/sec over the network. > > My setup: > Running latest Solaris 10: # uname -a > SunOS solssd01 5.10 Generic_142901-02 i86pc i386 i86pc > > # zpool status DATA > ?pool: DATA > ?state: ONLINE > config: > ? ? ? ?NAME ? ? ? ?STATE ? ? READ WRITE CKSUM > ? ? ? ?DATA ? ? ? ?ONLINE ? ? ? 0 ? ? 0 ? ? 0 > ? ? ? ? ?raidz1 ? ?ONLINE ? ? ? 0 ? ? 0 ? ? 0 > ? ? ? ? ? ?c2t5d0 ?ONLINE ? ? ? 0 ? ? 0 ? ? 0 > ? ? ? ? ? ?c2t4d0 ?ONLINE ? ? ? 0 ? ? 0 ? ? 0 > ? ? ? ? ? ?c2t3d0 ?ONLINE ? ? ? 0 ? ? 0 ? ? 0 > ? ? ? ? ? ?c2t2d0 ?ONLINE ? ? ? 0 ? ? 0 ? ? 0 > ? ? ? ?spares > ? ? ? ? ?c0t2d0 ? ?AVAIL > errors: No known data errors > > # zfs list -r DATA > NAME ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? USED ?AVAIL ?REFER ?MOUNTPOINT > DATA ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?3,78T ? 229G ?3,78T ?/DATA > > All of the drives in this pool are 1.5tb western digital green drives. I am not seeing any error messages in /var/adm/messages, and "fmdump -eV" shows no errors... ? However, I am seeing some soft faults in "iostat -eEn": > ?---- errors --- > ?s/w h/w trn tot device > ?2 ? 0 ? 0 ? 2 c0t0d0 > ?1 ? 0 ? 0 ? 1 c1t0d0 > ?2 ? 0 ? 0 ? 2 c2t1d0 > 151 ? 0 ? 0 151 c2t2d0 > 151 ? 0 ? 0 151 c2t3d0 > 153 ? 0 ? 0 153 c2t4d0 > 153 ? 0 ? 0 153 c2t5d0 > ?2 ? 0 ? 0 ? 2 c0t1d0 > ?3 ? 0 ? 0 ? 3 c0t2d0 > ?0 ? 0 ? 0 ? 0 solssd01:vold(pid531) > c0t0d0 ? ? ? ? ? Soft Errors: 2 Hard Errors: 0 Transport Errors: 0 > Vendor: Sun ? ? ?Product: STK RAID INT ? ? Revision: V1.0 Serial No: > Size: 31.87GB <31866224128 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > Illegal Request: 2 Predictive Failure Analysis: 0 > c1t0d0 ? ? ? ? ? Soft Errors: 1 Hard Errors: 0 Transport Errors: 0 > Vendor: _NEC ? ? Product: DVD_RW ND-3500AG Revision: 2.16 Serial No: > Size: 0.00GB <0 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > Illegal Request: 1 Predictive Failure Analysis: 0 > c2t1d0 ? ? ? ? ? Soft Errors: 2 Hard Errors: 0 Transport Errors: 0 > Vendor: ATA ? ? ?Product: SAMSUNG HD753LJ ?Revision: 1113 Serial No: > Size: 750.16GB <750156373504 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > Illegal Request: 2 Predictive Failure Analysis: 0 > c2t2d0 ? ? ? ? ? Soft Errors: 151 Hard Errors: 0 Transport Errors: 0 > Vendor: ATA ? ? ?Product: WDC WD15EADS-00R Revision: 0A01 Serial No: > Size: 1500.30GB <1500301909504 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > Illegal Request: 151 Predictive Failure Analysis: 0 > c2t3d0 ? ? ? ? ? Soft Errors: 151 Hard Errors: 0 Transport Errors: 0 > Vendor: ATA ? ? ?Product: WDC WD15EADS-00R Revision: 0A01 Serial No: > Size: 1500.30GB <1500301909504 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > Illegal Request: 151 Predictive Failure Analysis: 0 > c2t4d0 ? ? ? ? ? Soft Errors: 153 Hard Errors: 0 Transport Errors: 0 > Vendor: ATA ? ? ?Product: WDC WD15EADS-00R Revision: 0A01 Serial No: > Size: 1500.30GB <1500301909504 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > Illegal Request: 153 Predictive Failure Analysis: 0 > c2t5d0 ? ? ? ? ? Soft Errors: 153 Hard Errors: 0 Transport Errors: 0 > Vendor: ATA ? ? ?Product: WDC WD15EADS-00R Revision: 0A01 Serial No: > Size: 1500.30GB <1500301909504 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > Illegal Request: 153 Predictive Failure Analysis: 0 > c0t1d0 ? ? ? ? ? Soft Errors: 2 Hard Errors: 0 Transport Errors: 0 > Vendor: Sun ? ? ?Product: STK RAID INT ? ? Revision: V1.0 Serial No: > Size: 31.87GB <31866224128 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > Illegal Request: 2 Predictive Failure Analysis: 0 > c0t2d0 ? ? ? ? ? Soft Errors: 3 Hard Errors: 0 Transport Errors: 0 > Vendor: Sun ? ? ?Product: STK RAID INT ? ? Revision: V1.0 Serial No: > Size: 1497.86GB <1497859358208 bytes> > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > Illegal Request: 3 Predictive Failure Analysis: 0 > > I am curious as to why the counter for "Illegal request" goes up all the time. ?The machine was rebooted ~11 hours ago, and it goes up all the time when I try to use the pool... > > The machine is a quite powerful one, and top shows no cpu load, no iowait and plenty of available memory. ?The machine basicly doesn''t do anything at the moment, still it can take several minutes to copy a 300mb file from somewhere in the pool to /tmp/... > # top > last pid: ?1383; ?load avg: ?0.01, ?0.00, ?0.00; ?up 0+10:47:57 ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?01:39:17 > 55 processes: 54 sleeping, 1 on cpu > CPU states: 99.0% idle, ?0.0% user, ?1.0% kernel, ?0.0% iowait, ?0.0% swap > Kernel: 193 ctxsw, 3 trap, 439 intr, 298 syscall, 3 flt > Memory: 8186M phys mem, 4699M free mem, 2048M total swap, 2048M free swap > > I thought I might have run into problems described here on the forums with the ARC and fragmentation, but it doesn''t seem so: > # echo "::arc"|mdb -k > hits ? ? ? ? ? ? ? ? ? ? ?= ? ?490044 > misses ? ? ? ? ? ? ? ? ? ?= ? ? 37004 > demand_data_hits ? ? ? ? ?= ? ?282392 > demand_data_misses ? ? ? ?= ? ? ?2113 > demand_metadata_hits ? ? ?= ? ?191757 > demand_metadata_misses ? ?= ? ? 21034 > prefetch_data_hits ? ? ? ?= ? ? ? 851 > prefetch_data_misses ? ? ?= ? ? 10265 > prefetch_metadata_hits ? ?= ? ? 15044 > prefetch_metadata_misses ?= ? ? ?3592 > mru_hits ? ? ? ? ? ? ? ? ?= ? ? 73416 > mru_ghost_hits ? ? ? ? ? ?= ? ? ? ?16 > mfu_hits ? ? ? ? ? ? ? ? ?= ? ?401500 > mfu_ghost_hits ? ? ? ? ? ?= ? ? ? ?24 > deleted ? ? ? ? ? ? ? ? ? = ? ? ?1555 > recycle_miss ? ? ? ? ? ? ?= ? ? ? ? 0 > mutex_miss ? ? ? ? ? ? ? ?= ? ? ? ? 0 > evict_skip ? ? ? ? ? ? ? ?= ? ? ?1487 > hash_elements ? ? ? ? ? ? = ? ? 37032 > hash_elements_max ? ? ? ? = ? ? 37045 > hash_collisions ? ? ? ? ? = ? ? 10094 > hash_chains ? ? ? ? ? ? ? = ? ? ?4365 > hash_chain_max ? ? ? ? ? ?= ? ? ? ? 4 > p ? ? ? ? ? ? ? ? ? ? ? ? = ? ? ?3576 MB > c ? ? ? ? ? ? ? ? ? ? ? ? = ? ? ?7154 MB > c_min ? ? ? ? ? ? ? ? ? ? = ? ? ? 894 MB > c_max ? ? ? ? ? ? ? ? ? ? = ? ? ?7154 MB > size ? ? ? ? ? ? ? ? ? ? ?= ? ? ?1797 MB > hdr_size ? ? ? ? ? ? ? ? ?= ? 8002680 > data_size ? ? ? ? ? ? ? ? = 1866272256 > other_size ? ? ? ? ? ? ? ?= ?10519712 > l2_hits ? ? ? ? ? ? ? ? ? = ? ? ? ? 0 > l2_misses ? ? ? ? ? ? ? ? = ? ? ? ? 0 > l2_feeds ? ? ? ? ? ? ? ? ?= ? ? ? ? 0 > l2_rw_clash ? ? ? ? ? ? ? = ? ? ? ? 0 > l2_read_bytes ? ? ? ? ? ? = ? ? ? ? 0 > l2_write_bytes ? ? ? ? ? ?= ? ? ? ? 0 > l2_writes_sent ? ? ? ? ? ?= ? ? ? ? 0 > l2_writes_done ? ? ? ? ? ?= ? ? ? ? 0 > l2_writes_error ? ? ? ? ? = ? ? ? ? 0 > l2_writes_hdr_miss ? ? ? ?= ? ? ? ? 0 > l2_evict_lock_retry ? ? ? = ? ? ? ? 0 > l2_evict_reading ? ? ? ? ?= ? ? ? ? 0 > l2_free_on_write ? ? ? ? ?= ? ? ? ? 0 > l2_abort_lowmem ? ? ? ? ? = ? ? ? ? 0 > l2_cksum_bad ? ? ? ? ? ? ?= ? ? ? ? 0 > l2_io_error ? ? ? ? ? ? ? = ? ? ? ? 0 > l2_size ? ? ? ? ? ? ? ? ? = ? ? ? ? 0 > l2_hdr_size ? ? ? ? ? ? ? = ? ? ? ? 0 > memory_throttle_count ? ? = ? ? ? ? 0 > arc_no_grow ? ? ? ? ? ? ? = ? ? ? ? 0 > arc_tempreserve ? ? ? ? ? = ? ? ? ? 0 MB > arc_meta_used ? ? ? ? ? ? = ? ? ? 372 MB > arc_meta_limit ? ? ? ? ? ?= ? ? ?1788 MB > arc_meta_max ? ? ? ? ? ? ?= ? ? ? 372 MB > > I then tried to start a scrub, and it seems like it will take forever... It used to take a few hours, now it says it will be done in almost 700 hours: > ?scrub: scrub in progress for 4h43m, 0,68% done, 685h2m to go > > Does anyone have any clue as to what is happening, and what I can do? ?If a disk is failing without the OS noticing, it would be nice to find a way to know which drive it is, and get it exchanged before it''s too late. > > All help is apreciated... > > Yours sincerly, > Morten-Christian Bernson > -- > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
Morten-Christian Bernson
2009-Dec-28 12:53 UTC
[zfs-discuss] Extremely bad performance - hw failure?
> The best place to start looking at disk-related > performance problems > is iostat. > Slow disks will show high service times. There are > many options, but I > usually use something like: > iostat -zxcnPT d 1 > > Ignore the first line. Look at the service times. > They should be > elow 10ms > for good performance.Thanks for the reply! It seems that c2t4d0s0 is the one making trouble, as I suspected. Here is a sample line from the output while copying a file from the DATA-pool to /tmp/: r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 17.0 6.0 896.5 8.5 0.7 27.6 31.8 1201.5 73 100 c2t4d0s0 The asvc_t is insanely high, while the other disks are from 0 to max 80, but mostly around 0. What does this mean? Is the disk defective? Could I get it replaced by the warranty? If I want to exchange it in the pool with the hot spare, how do I go around doing that? I want to make sure that the whole pool doesn''t collapse on me if I take out this disk, and then the problems isn''t really this disk but some other, and the whole pool is gone... -- This message posted from opensolaris.org
Richard Elling
2009-Dec-28 16:29 UTC
[zfs-discuss] Extremely bad performance - hw failure?
On Dec 27, 2009, at 10:21 PM, Joe Little wrote:> I''ve had this happen to me too. I found some dtrace scripts at the > time that showed that the file system was spending too much time > finding available 128k blocks or the like as I was near full per each > disk, even though combined I still had 140GB left of my 3TB pool. The > SPA code I believe it was was spending too much time walking the > available pool for continguous space for new writes, and this > affecting both read and write performance dramatically (measured in > kb/sec). > > I was able to alleviate the pressure so to speak by adjusting the > recordsize for the pool down to 8k (32k is likely more recommended) > and from there I could then start to clear out space. Anything below > 10% available space seems to cause ZFS to start behaving poorly, and > getting down lower increases the problems. But the root cause was > metadata management on pools w/ less than 5-10% disk space left.The better solution is to use b129 or later, where CR 6869229, zfs should switch to shiny new metaslabs more frequently, was integrated. http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6869229 -- richard> In my case, I had lots of symlinks, lots of small files, and also > dozens of snapshots. My pool was a RAID10 (aka, 3 mirror sets > striped). > > > On Sun, Dec 27, 2009 at 4:52 PM, Morten-Christian Bernson > <mcb at uib.no> wrote: >> Lately my zfs pool in my home server has degraded to a state where >> it can be said it doesn''t work at all. Read spead is slower than I >> can read from the internet on my slow dsl-line... This is compared >> to just a short while ago, where I could read from it with over >> 50mb/sec over the network. >> >> My setup: >> Running latest Solaris 10: # uname -a >> SunOS solssd01 5.10 Generic_142901-02 i86pc i386 i86pc >> >> # zpool status DATA >> pool: DATA >> state: ONLINE >> config: >> NAME STATE READ WRITE CKSUM >> DATA ONLINE 0 0 0 >> raidz1 ONLINE 0 0 0 >> c2t5d0 ONLINE 0 0 0 >> c2t4d0 ONLINE 0 0 0 >> c2t3d0 ONLINE 0 0 0 >> c2t2d0 ONLINE 0 0 0 >> spares >> c0t2d0 AVAIL >> errors: No known data errors >> >> # zfs list -r DATA >> NAME USED AVAIL REFER MOUNTPOINT >> DATA 3,78T 229G 3,78T /DATA >> >> All of the drives in this pool are 1.5tb western digital green >> drives. I am not seeing any error messages in /var/adm/messages, >> and "fmdump -eV" shows no errors... However, I am seeing some >> soft faults in "iostat -eEn": >> ---- errors --- >> s/w h/w trn tot device >> 2 0 0 2 c0t0d0 >> 1 0 0 1 c1t0d0 >> 2 0 0 2 c2t1d0 >> 151 0 0 151 c2t2d0 >> 151 0 0 151 c2t3d0 >> 153 0 0 153 c2t4d0 >> 153 0 0 153 c2t5d0 >> 2 0 0 2 c0t1d0 >> 3 0 0 3 c0t2d0 >> 0 0 0 0 solssd01:vold(pid531) >> c0t0d0 Soft Errors: 2 Hard Errors: 0 Transport Errors: 0 >> Vendor: Sun Product: STK RAID INT Revision: V1.0 Serial No: >> Size: 31.87GB <31866224128 bytes> >> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 >> Illegal Request: 2 Predictive Failure Analysis: 0 >> c1t0d0 Soft Errors: 1 Hard Errors: 0 Transport Errors: 0 >> Vendor: _NEC Product: DVD_RW ND-3500AG Revision: 2.16 Serial No: >> Size: 0.00GB <0 bytes> >> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 >> Illegal Request: 1 Predictive Failure Analysis: 0 >> c2t1d0 Soft Errors: 2 Hard Errors: 0 Transport Errors: 0 >> Vendor: ATA Product: SAMSUNG HD753LJ Revision: 1113 Serial No: >> Size: 750.16GB <750156373504 bytes> >> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 >> Illegal Request: 2 Predictive Failure Analysis: 0 >> c2t2d0 Soft Errors: 151 Hard Errors: 0 Transport Errors: 0 >> Vendor: ATA Product: WDC WD15EADS-00R Revision: 0A01 Serial No: >> Size: 1500.30GB <1500301909504 bytes> >> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 >> Illegal Request: 151 Predictive Failure Analysis: 0 >> c2t3d0 Soft Errors: 151 Hard Errors: 0 Transport Errors: 0 >> Vendor: ATA Product: WDC WD15EADS-00R Revision: 0A01 Serial No: >> Size: 1500.30GB <1500301909504 bytes> >> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 >> Illegal Request: 151 Predictive Failure Analysis: 0 >> c2t4d0 Soft Errors: 153 Hard Errors: 0 Transport Errors: 0 >> Vendor: ATA Product: WDC WD15EADS-00R Revision: 0A01 Serial No: >> Size: 1500.30GB <1500301909504 bytes> >> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 >> Illegal Request: 153 Predictive Failure Analysis: 0 >> c2t5d0 Soft Errors: 153 Hard Errors: 0 Transport Errors: 0 >> Vendor: ATA Product: WDC WD15EADS-00R Revision: 0A01 Serial No: >> Size: 1500.30GB <1500301909504 bytes> >> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 >> Illegal Request: 153 Predictive Failure Analysis: 0 >> c0t1d0 Soft Errors: 2 Hard Errors: 0 Transport Errors: 0 >> Vendor: Sun Product: STK RAID INT Revision: V1.0 Serial No: >> Size: 31.87GB <31866224128 bytes> >> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 >> Illegal Request: 2 Predictive Failure Analysis: 0 >> c0t2d0 Soft Errors: 3 Hard Errors: 0 Transport Errors: 0 >> Vendor: Sun Product: STK RAID INT Revision: V1.0 Serial No: >> Size: 1497.86GB <1497859358208 bytes> >> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 >> Illegal Request: 3 Predictive Failure Analysis: 0 >> >> I am curious as to why the counter for "Illegal request" goes up >> all the time. The machine was rebooted ~11 hours ago, and it goes >> up all the time when I try to use the pool... >> >> The machine is a quite powerful one, and top shows no cpu load, no >> iowait and plenty of available memory. The machine basicly doesn''t >> do anything at the moment, still it can take several minutes to >> copy a 300mb file from somewhere in the pool to /tmp/... >> # top >> last pid: 1383; load avg: 0.01, 0.00, 0.00; up >> 0 >> + >> 10 >> : >> 47 >> : >> 57 >> 01:39:17 >> 55 processes: 54 sleeping, 1 on cpu >> CPU states: 99.0% idle, 0.0% user, 1.0% kernel, 0.0% iowait, >> 0.0% swap >> Kernel: 193 ctxsw, 3 trap, 439 intr, 298 syscall, 3 flt >> Memory: 8186M phys mem, 4699M free mem, 2048M total swap, 2048M >> free swap >> >> I thought I might have run into problems described here on the >> forums with the ARC and fragmentation, but it doesn''t seem so: >> # echo "::arc"|mdb -k >> hits = 490044 >> misses = 37004 >> demand_data_hits = 282392 >> demand_data_misses = 2113 >> demand_metadata_hits = 191757 >> demand_metadata_misses = 21034 >> prefetch_data_hits = 851 >> prefetch_data_misses = 10265 >> prefetch_metadata_hits = 15044 >> prefetch_metadata_misses = 3592 >> mru_hits = 73416 >> mru_ghost_hits = 16 >> mfu_hits = 401500 >> mfu_ghost_hits = 24 >> deleted = 1555 >> recycle_miss = 0 >> mutex_miss = 0 >> evict_skip = 1487 >> hash_elements = 37032 >> hash_elements_max = 37045 >> hash_collisions = 10094 >> hash_chains = 4365 >> hash_chain_max = 4 >> p = 3576 MB >> c = 7154 MB >> c_min = 894 MB >> c_max = 7154 MB >> size = 1797 MB >> hdr_size = 8002680 >> data_size = 1866272256 >> other_size = 10519712 >> l2_hits = 0 >> l2_misses = 0 >> l2_feeds = 0 >> l2_rw_clash = 0 >> l2_read_bytes = 0 >> l2_write_bytes = 0 >> l2_writes_sent = 0 >> l2_writes_done = 0 >> l2_writes_error = 0 >> l2_writes_hdr_miss = 0 >> l2_evict_lock_retry = 0 >> l2_evict_reading = 0 >> l2_free_on_write = 0 >> l2_abort_lowmem = 0 >> l2_cksum_bad = 0 >> l2_io_error = 0 >> l2_size = 0 >> l2_hdr_size = 0 >> memory_throttle_count = 0 >> arc_no_grow = 0 >> arc_tempreserve = 0 MB >> arc_meta_used = 372 MB >> arc_meta_limit = 1788 MB >> arc_meta_max = 372 MB >> >> I then tried to start a scrub, and it seems like it will take >> forever... It used to take a few hours, now it says it will be done >> in almost 700 hours: >> scrub: scrub in progress for 4h43m, 0,68% done, 685h2m to go >> >> Does anyone have any clue as to what is happening, and what I can >> do? If a disk is failing without the OS noticing, it would be nice >> to find a way to know which drive it is, and get it exchanged >> before it''s too late. >> >> All help is apreciated... >> >> Yours sincerly, >> Morten-Christian Bernson >> -- >> This message posted from opensolaris.org >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Richard Elling
2009-Dec-28 16:32 UTC
[zfs-discuss] Extremely bad performance - hw failure?
On Dec 28, 2009, at 4:53 AM, Morten-Christian Bernson wrote:>> The best place to start looking at disk-related >> performance problems >> is iostat. >> Slow disks will show high service times. There are >> many options, but I >> usually use something like: >> iostat -zxcnPT d 1 >> >> Ignore the first line. Look at the service times. >> They should be >> elow 10ms >> for good performance. > > Thanks for the reply! > > It seems that c2t4d0s0 is the one making trouble, as I suspected. > Here is a sample line from the output while copying a file from the > DATA-pool to /tmp/: > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 17.0 6.0 896.5 8.5 0.7 27.6 31.8 1201.5 73 100 c2t4d0s0 > The asvc_t is insanely high, while the other disks are from 0 to max > 80, but mostly around 0.Is someone yelling at it? :-) But seriously, check the drive for vibration and if that doesn''t fix the performance, consider replacing it or the cables. 1.2 seconds is just way too slow for an I/O request.> > What does this mean? Is the disk defective? Could I get it > replaced by the warranty? If I want to exchange it in the pool with > the hot spare, how do I go around doing that? I want to make sure > that the whole pool doesn''t collapse on me if I take out this disk, > and then the problems isn''t really this disk but some other, and the > whole pool is gone...With a 1.2 second response time, the entire pool will be really, reaallllltyyyyy sllllllllllllllooooooooooooowwwwwwwwwwwwwww. You should see something more like 10ms response time -- two orders of magnitude better. -- richard>
Morten-Christian Bernson
2009-Dec-28 22:01 UTC
[zfs-discuss] Extremely bad performance - hw failure?
I tried changing cables, nothing changed. Then I saw something about smartmontools on the net, and I installed that. It reported that c2t4d0 had indeed reported a SMART error. Why didn''t Solaris detect this? I then went ahead and offlined c2t4d0, after which the pool performed much more like expected with excellent speeds. I tried to do a zpool replace with the hotspare, but it was connected through another raid controller which seems to have taken some of the sectors to itself, which lead to the disk being too small to be able to replace the defective, even though they are identical... This was a good thing to discover before it was too late as well :) I then physically replaced the defective drive with the hotspare, and then the resilvering could begin. If everything goes well, maybe it will be in normal working condition again in the morning :) Thanks to everyone for helping out. -- This message posted from opensolaris.org
Richard Elling
2009-Dec-28 23:26 UTC
[zfs-discuss] Extremely bad performance - hw failure?
On Dec 28, 2009, at 2:01 PM, Morten-Christian Bernson wrote:> I tried changing cables, nothing changed. Then I saw something > about smartmontools on the net, and I installed that. It reported > that c2t4d0 had indeed reported a SMART error. Why didn''t Solaris > detect this?In your OP, I see: c2t4d0 Soft Errors: 153> I then went ahead and offlined c2t4d0, after which the pool > performed much more like expected with excellent speeds.good!> I tried to do a zpool replace with the hotspare, but it was > connected through another raid controller which seems to have taken > some of the sectors to itself, which lead to the disk being too > small to be able to replace the defective, even though they are > identical... This was a good thing to discover before it was too > late as well :)In later builds, a few sectors here or there will be accommodated.> > I then physically replaced the defective drive with the hotspare, > and then the resilvering could begin. If everything goes well, > maybe it will be in normal working condition again in the morning :) > > Thanks to everyone for helping out.Good luck! -- richard