Robert Milkowski
2006-Oct-26 11:42 UTC
[zfs-discuss] zpool status - very slow during heavy IOs
Hi.
When there''s a lot of IOs to a pool then zpool status is really
slow.
These are just statistics and it should work quick.
# truss -ED zpool status
[...]
0.0008 0.0001 fstat64(1, 0xFFBFAC40) = 0
pool: nfs-1
0.0015 0.0001 write(1, " p o o l : n f s -".., 14) = 14
state: ONLINE
0.0013 0.0001 write(1, " s t a t e : O N L I".., 15) = 15
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
0.0008 0.0001 write(1, " s t a t u s : O n e ".., 142) = 142
action: Determine if the device needs to be replaced, and clear the errors
using ''zpool clear'' or replace the device with
''zpool replace''.
0.0003 0.0001 write(1, " a c t i o n : D e t e".., 140) = 140
see: http://www.sun.com/msg/ZFS-8000-9P
0.0006 0.0003 write(1, " s e e : h t t p".., 43) = 43
0.0013 0.0000 time() = 1161862691
scrub: scrub in progress, 23.16% done, 0h34m to go
0.0012 0.0001 write(1, " s c r u b : s c r u".., 52) = 52
[ so here we''ve got summary which was showed quickly]
5.1836 0.0018 open("/dev/dsk/c3t0d0s0", O_RDONLY) = 4
0.0008 0.0000 fstat(4, 0xFFBFB844) = 0
0.0005 0.0001 modctl(MODSIZEOF_DEVID, 0x00800008, 0xFFBFB8CC, 0x0003C714,
0xFF24E4F8) = 0
0.0004 0.0000 modctl(MODGETDEVID, 0x00800008, 0x0000002C, 0x0010E130,
0xFF24E4F8) = 0
0.0002 0.0000 fstat(4, 0xFFBFB844) = 0
0.0003 0.0000 modctl(MODSIZEOF_MINORNAME, 0x00800008, 0x00006000, 0xFFBFB8CC,
0xFF24E4F8) = 0
0.0003 0.0001 modctl(MODGETMINORNAME, 0x00800008, 0x00006000, 0x00000002,
0x00129FC8) = 0
0.0004 0.0000 close(4) = 0
3.1607 0.0015 open("/dev/dsk/c2t0d0s0", O_RDONLY) = 4
0.0020 0.0000 fstat(4, 0xFFBFB844) = 0
0.0003 0.0001 modctl(MODSIZEOF_DEVID, 0x00800028, 0xFFBFB8CC, 0x0003C714,
0xFF24D6A4) = 0
0.0003 0.0000 modctl(MODGETDEVID, 0x00800028, 0x00000030, 0x0010E130,
0xFF24D6A4) = 0
0.0003 0.0000 fstat(4, 0xFFBFB844) = 0
0.0003 0.0001 modctl(MODSIZEOF_MINORNAME, 0x00800028, 0x00006000, 0xFFBFB8CC,
0xFF24D6A4) = 0
0.0014 0.0001 modctl(MODGETMINORNAME, 0x00800028, 0x00006000, 0x00000002,
0x00129FC8) = 0
0.0006 0.0000 close(4) = 0
4.7857 0.0022 open("/dev/dsk/c3t1d0s0", O_RDONLY) = 4
0.0018 0.0000 fstat(4, 0xFFBFB844) = 0
0.0004 0.0001 modctl(MODSIZEOF_DEVID, 0x00800010, 0xFFBFB8CC, 0x0003C714,
0xFF24E4F8) = 0
0.0003 0.0000 modctl(MODGETDEVID, 0x00800010, 0x0000002C, 0x0010E130,
0xFF24E4F8) = 0
0.0003 0.0000 fstat(4, 0xFFBFB844) = 0
0.0003 0.0000 modctl(MODSIZEOF_MINORNAME, 0x00800010, 0x00006000, 0xFFBFB8CC,
0xFF24E4F8) = 0
0.0003 0.0000 modctl(MODGETMINORNAME, 0x00800010, 0x00006000, 0x00000002,
0x00129FC8) = 0
0.0003 0.0000 close(4) = 0
This message posted from opensolaris.org
Eric Schrock
2006-Oct-26 16:15 UTC
[zfs-discuss] zpool status - very slow during heavy IOs
See: 6430480 grabbing config lock as writer during I/O load can take excessively long - Eric On Thu, Oct 26, 2006 at 04:42:00AM -0700, Robert Milkowski wrote:> Hi. > > > When there''s a lot of IOs to a pool then zpool status is really slow. > These are just statistics and it should work quick. > > # truss -ED zpool status > [...] > 0.0008 0.0001 fstat64(1, 0xFFBFAC40) = 0 > pool: nfs-1 > 0.0015 0.0001 write(1, " p o o l : n f s -".., 14) = 14 > state: ONLINE > 0.0013 0.0001 write(1, " s t a t e : O N L I".., 15) = 15 > status: One or more devices has experienced an unrecoverable error. An > attempt was made to correct the error. Applications are unaffected. > 0.0008 0.0001 write(1, " s t a t u s : O n e ".., 142) = 142 > action: Determine if the device needs to be replaced, and clear the errors > using ''zpool clear'' or replace the device with ''zpool replace''. > 0.0003 0.0001 write(1, " a c t i o n : D e t e".., 140) = 140 > see: http://www.sun.com/msg/ZFS-8000-9P > 0.0006 0.0003 write(1, " s e e : h t t p".., 43) = 43 > 0.0013 0.0000 time() = 1161862691 > scrub: scrub in progress, 23.16% done, 0h34m to go > 0.0012 0.0001 write(1, " s c r u b : s c r u".., 52) = 52 > > [ so here we''ve got summary which was showed quickly] > > 5.1836 0.0018 open("/dev/dsk/c3t0d0s0", O_RDONLY) = 4 > 0.0008 0.0000 fstat(4, 0xFFBFB844) = 0 > 0.0005 0.0001 modctl(MODSIZEOF_DEVID, 0x00800008, 0xFFBFB8CC, 0x0003C714, 0xFF24E4F8) = 0 > 0.0004 0.0000 modctl(MODGETDEVID, 0x00800008, 0x0000002C, 0x0010E130, 0xFF24E4F8) = 0 > 0.0002 0.0000 fstat(4, 0xFFBFB844) = 0 > 0.0003 0.0000 modctl(MODSIZEOF_MINORNAME, 0x00800008, 0x00006000, 0xFFBFB8CC, 0xFF24E4F8) = 0 > 0.0003 0.0001 modctl(MODGETMINORNAME, 0x00800008, 0x00006000, 0x00000002, 0x00129FC8) = 0 > 0.0004 0.0000 close(4) = 0 > > > > 3.1607 0.0015 open("/dev/dsk/c2t0d0s0", O_RDONLY) = 4 > 0.0020 0.0000 fstat(4, 0xFFBFB844) = 0 > 0.0003 0.0001 modctl(MODSIZEOF_DEVID, 0x00800028, 0xFFBFB8CC, 0x0003C714, 0xFF24D6A4) = 0 > 0.0003 0.0000 modctl(MODGETDEVID, 0x00800028, 0x00000030, 0x0010E130, 0xFF24D6A4) = 0 > 0.0003 0.0000 fstat(4, 0xFFBFB844) = 0 > 0.0003 0.0001 modctl(MODSIZEOF_MINORNAME, 0x00800028, 0x00006000, 0xFFBFB8CC, 0xFF24D6A4) = 0 > 0.0014 0.0001 modctl(MODGETMINORNAME, 0x00800028, 0x00006000, 0x00000002, 0x00129FC8) = 0 > 0.0006 0.0000 close(4) = 0 > > 4.7857 0.0022 open("/dev/dsk/c3t1d0s0", O_RDONLY) = 4 > 0.0018 0.0000 fstat(4, 0xFFBFB844) = 0 > 0.0004 0.0001 modctl(MODSIZEOF_DEVID, 0x00800010, 0xFFBFB8CC, 0x0003C714, 0xFF24E4F8) = 0 > 0.0003 0.0000 modctl(MODGETDEVID, 0x00800010, 0x0000002C, 0x0010E130, 0xFF24E4F8) = 0 > 0.0003 0.0000 fstat(4, 0xFFBFB844) = 0 > 0.0003 0.0000 modctl(MODSIZEOF_MINORNAME, 0x00800010, 0x00006000, 0xFFBFB8CC, 0xFF24E4F8) = 0 > 0.0003 0.0000 modctl(MODGETMINORNAME, 0x00800010, 0x00006000, 0x00000002, 0x00129FC8) = 0 > 0.0003 0.0000 close(4) = 0 > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock
eric kustarz
2006-Oct-26 16:35 UTC
[zfs-discuss] zpool status - very slow during heavy IOs
Robert Milkowski wrote:> Hi. > > > When there''s a lot of IOs to a pool then zpool status is really slow. > These are just statistics and it should work quick.This is: 6430480 grabbing config lock as writer during I/O load can take excessively long eric> > # truss -ED zpool status > [...] > 0.0008 0.0001 fstat64(1, 0xFFBFAC40) = 0 > pool: nfs-1 > 0.0015 0.0001 write(1, " p o o l : n f s -".., 14) = 14 > state: ONLINE > 0.0013 0.0001 write(1, " s t a t e : O N L I".., 15) = 15 > status: One or more devices has experienced an unrecoverable error. An > attempt was made to correct the error. Applications are unaffected. > 0.0008 0.0001 write(1, " s t a t u s : O n e ".., 142) = 142 > action: Determine if the device needs to be replaced, and clear the errors > using ''zpool clear'' or replace the device with ''zpool replace''. > 0.0003 0.0001 write(1, " a c t i o n : D e t e".., 140) = 140 > see: http://www.sun.com/msg/ZFS-8000-9P > 0.0006 0.0003 write(1, " s e e : h t t p".., 43) = 43 > 0.0013 0.0000 time() = 1161862691 > scrub: scrub in progress, 23.16% done, 0h34m to go > 0.0012 0.0001 write(1, " s c r u b : s c r u".., 52) = 52 > > [ so here we''ve got summary which was showed quickly] > > 5.1836 0.0018 open("/dev/dsk/c3t0d0s0", O_RDONLY) = 4 > 0.0008 0.0000 fstat(4, 0xFFBFB844) = 0 > 0.0005 0.0001 modctl(MODSIZEOF_DEVID, 0x00800008, 0xFFBFB8CC, 0x0003C714, 0xFF24E4F8) = 0 > 0.0004 0.0000 modctl(MODGETDEVID, 0x00800008, 0x0000002C, 0x0010E130, 0xFF24E4F8) = 0 > 0.0002 0.0000 fstat(4, 0xFFBFB844) = 0 > 0.0003 0.0000 modctl(MODSIZEOF_MINORNAME, 0x00800008, 0x00006000, 0xFFBFB8CC, 0xFF24E4F8) = 0 > 0.0003 0.0001 modctl(MODGETMINORNAME, 0x00800008, 0x00006000, 0x00000002, 0x00129FC8) = 0 > 0.0004 0.0000 close(4) = 0 > > > > 3.1607 0.0015 open("/dev/dsk/c2t0d0s0", O_RDONLY) = 4 > 0.0020 0.0000 fstat(4, 0xFFBFB844) = 0 > 0.0003 0.0001 modctl(MODSIZEOF_DEVID, 0x00800028, 0xFFBFB8CC, 0x0003C714, 0xFF24D6A4) = 0 > 0.0003 0.0000 modctl(MODGETDEVID, 0x00800028, 0x00000030, 0x0010E130, 0xFF24D6A4) = 0 > 0.0003 0.0000 fstat(4, 0xFFBFB844) = 0 > 0.0003 0.0001 modctl(MODSIZEOF_MINORNAME, 0x00800028, 0x00006000, 0xFFBFB8CC, 0xFF24D6A4) = 0 > 0.0014 0.0001 modctl(MODGETMINORNAME, 0x00800028, 0x00006000, 0x00000002, 0x00129FC8) = 0 > 0.0006 0.0000 close(4) = 0 > > 4.7857 0.0022 open("/dev/dsk/c3t1d0s0", O_RDONLY) = 4 > 0.0018 0.0000 fstat(4, 0xFFBFB844) = 0 > 0.0004 0.0001 modctl(MODSIZEOF_DEVID, 0x00800010, 0xFFBFB8CC, 0x0003C714, 0xFF24E4F8) = 0 > 0.0003 0.0000 modctl(MODGETDEVID, 0x00800010, 0x0000002C, 0x0010E130, 0xFF24E4F8) = 0 > 0.0003 0.0000 fstat(4, 0xFFBFB844) = 0 > 0.0003 0.0000 modctl(MODSIZEOF_MINORNAME, 0x00800010, 0x00006000, 0xFFBFB8CC, 0xFF24E4F8) = 0 > 0.0003 0.0000 modctl(MODGETMINORNAME, 0x00800010, 0x00006000, 0x00000002, 0x00129FC8) = 0 > 0.0003 0.0000 close(4) = 0 > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss