Carsten Aulbert
2011-Sep-15 09:13 UTC
[zfs-discuss] file system under heavy load, how to find out what the cause is?
Hi
strange problem on a x4440:
pool: rpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t5000CCA00A3E1D98d0s0 ONLINE 0 0 0
c1t5000CCA00A3D1BE4d0s0 ONLINE 0 0 0
errors: No known data errors
root at x4440 # zpool iostat -v rpool 15 2
capacity operations bandwidth
pool alloc free read write read write
--------------------------- ----- ----- ----- ----- ----- -----
rpool 65.5G 70.5G 0 333 55.5K 35.4M
mirror 65.5G 70.5G 0 333 55.5K 35.4M
c1t5000CCA00A3E1D98d0s0 - - 0 300 28.3K 35.4M
c1t5000CCA00A3D1BE4d0s0 - - 0 300 28.3K 35.4M
--------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
--------------------------- ----- ----- ----- ----- ----- -----
rpool 65.5G 70.5G 0 643 0 65.8M
mirror 65.5G 70.5G 0 643 0 65.8M
c1t5000CCA00A3E1D98d0s0 - - 0 559 0 65.8M
c1t5000CCA00A3D1BE4d0s0 - - 0 557 0 65.7M
--------------------------- ----- ----- ----- ----- ----- -----
(2 10k SAS drives)
iosnoop does not really tell me something useful:
root at x4440 # ./iosnoop -d sd32 -d sd33 -A | head
TIME STIME DELTA DEVICE INS MAJ MIN UID PID PPID D BLOCK SIZE MOUNT FILE PATH
COMM ARGS
1812818090420 1812818079998 10421 sd33 33 32 2112 0 5 0 W 183388660 131072
<none> <none> <none> zpool-rpool zpool-rpool\0
1812818091454 1812818082162 9292 sd33 33 32 2112 0 5 0 W 183388916 131072
<none> <none> <none> zpool-rpool zpool-rpool\0
1812818098863 1812818072105 26757 sd33 33 32 2112 0 5 0 W 183386612 131072
<none> <none> <none> zpool-rpool zpool-rpool\0
1812818099938 1812818073155 26783 sd33 33 32 2112 0 5 0 W 183386868 131072
<none> <none> <none> zpool-rpool zpool-rpool\0
1812818101468 1812818074220 27247 sd33 33 32 2112 0 5 0 W 183387124 131072
<none> <none> <none> zpool-rpool zpool-rpool\0
1812818102540 1812818075278 27262 sd33 33 32 2112 0 5 0 W 183387380 131072
<none> <none> <none> zpool-rpool zpool-rpool\0
1812818103615 1812818076822 26792 sd33 33 32 2112 0 5 0 W 183387636 131072
<none> <none> <none> zpool-rpool zpool-rpool\0
1812818104689 1812818077894 26794 sd33 33 32 2112 0 5 0 W 183387892 131072
<none> <none> <none> zpool-rpool zpool-rpool\0
1812818105738 1812818078971 26767 sd33 33 32 2112 0 5 0 W 183388148 131072
<none> <none> <none> zpool-rpool zpool-rpool\0
PID 5 is just
root 5 0 0 Aug 25 ? 3102:27 zpool-rpool
to round it up:
root at x4440 # ./arc_summary.pl
System Memory:
Physical RAM: 131063 MB
Free Memory : 13430 MB
LotsFree: 1991 MB
ZFS Tunables (/etc/system):
ARC Size:
Current Size: 89931 MB (arcsize)
Target Size (Adaptive): 97854 MB (c)
Min Size (Hard Limit): 16254 MB (zfs_arc_min)
Max Size (Hard Limit): 130039 MB (zfs_arc_max)
ARC Size Breakdown:
Most Recently Used Cache Size: 99% 97296 MB (p)
Most Frequently Used Cache Size: 0% 557 MB (c-p)
ARC Efficency:
Cache Access Total: 942093365
Cache Hit Ratio: 99% 941013507 [Defined State for
buffer]
Cache Miss Ratio: 0% 1079858 [Undefined State for
Buffer]
REAL Hit Ratio: 96% 908987855 [MRU/MFU Hits Only]
Data Demand Efficiency: 99%
Data Prefetch Efficiency: 73%
CACHE HITS BY CACHE LIST:
Anon: 3% 31367306 [ New
Customer, First Cache Hit ]
Most Recently Used: 44% 417379490 (mru) [
Return Customer ]
Most Frequently Used: 52% 491608365 (mfu) [
Frequent Customer ]
Most Recently Used Ghost: 0% 635995 (mru_ghost) [
Return Customer Evicted, Now Back ]
Most Frequently Used Ghost: 0% 22351 (mfu_ghost) [
Frequent Customer Evicted, Now Back ]
CACHE HITS BY DATA TYPE:
Demand Data: 91% 858469425
Prefetch Data: 0% 2843857
Demand Metadata: 5% 48740552
Prefetch Metadata: 3% 30959673
CACHE MISSES BY DATA TYPE:
Demand Data: 2% 24982
Prefetch Data: 94% 1023544
Demand Metadata: 1% 16042
Prefetch Metadata: 1% 15290
---------------------------------------------
Has nayone any idea what''s going on here?
Cheers
carsten
--
Dr. Carsten Aulbert - Max Planck Institute for Gravitational Physics
Callinstrasse 38, 30167 Hannover, Germany
Phone/Fax: +49 511 762-17185 / -17193
http://www.top500.org/system/9234 | http://www.top500.org/connfam/6
CaCert Assurer | Get free certificates from http://www.cacert.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1871 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110915/946abc5f/attachment.bin>
Eric Sproul
2011-Sep-20 14:38 UTC
[zfs-discuss] file system under heavy load, how to find out what the cause is?
On Thu, Sep 15, 2011 at 5:13 AM, Carsten Aulbert <Carsten.Aulbert at aei.mpg.de> wrote:> Has nayone any idea what''s going on here?Carsten, It will be more visible at the VFS layer with fsstat. The following one-liner will pull out all ZFS filesystems and pass the list as arguments to fsstat so you can see activity broken down by mountpoint: fsstat -i `mount | awk ''{if($3 ~ /^[^\/:]+\//) {print $1;}}''` 1 Of course this only works for POSIX filesystems. This won''t catch activity to zvols. Maybe that won''t matter in your case. Eric