About a month ago, I upgraded my workstation to Nevada build 49. When
I ran "zpool status" on my existing zfs pool (a couple mirrored
partitions, one from each of the system''s two disks), I discovered that
the existing on-disk format was at version 1 and that I could upgrade
to version 3. I chose to defer doing so, until I was sure that I would
no longer boot back into the partition holding my previous root (from
build 29).
Last week, I decided it was time to take the plunge and invoked
% zpool upgrade zfs
The upgrade completed successfully, with no error messages.
But since that upgrade I''ve been getting unremitting bursts of disk
i/o. When there''s no activity on the system, they occur at a frequency
of 1/minute. (With activity, the frequency is higher. From listening
to my disks, there seems to be considerably more i/o traffic than
before the upgrade.) Here''s iostat info from when the system is
essentially idle:
% iostat 20
tty sd0 sd1 sd3 nfs1 cpu
tin tout kps tps serv kps tps serv kps tps serv kps tps serv us sy wt id
0 3 273 29 120 280 29 93 0 0 0 0 0 9 26 6 0 69
0 12 11 5 20 11 5 17 0 0 0 0 0 0 2 1 0 97
0 4 0 0 0 1 1 20 0 0 0 0 0 0 2 1 0 97
0 4 2 0 3 1 0 6 0 0 0 0 0 0 2 1 0 97
0 4 11 5 20 11 5 18 0 0 0 0 0 0 1 1 0 98
0 4 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 98
0 4 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 98
0 4 11 5 18 11 5 17 0 0 0 0 0 0 1 1 0 98
0 4 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 98
0 4 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 98
0 4 11 5 19 11 5 17 0 0 0 0 0 0 1 1 0 98
^C%
This periodic activity did not occur before I upgraded my zfs pool, and
I can''t think of any other change that might account for it. The pool
is completely healthy:
% zpool status zfs
pool: zfs
state: ONLINE
scrub: scrub completed with 0 errors on Fri Jan 12 15:34:05 2007
config:
NAME STATE READ WRITE CKSUM
zfs ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t1d0s6 ONLINE 0 0 0
c0t0d0s6 ONLINE 0 0 0
errors: No known data errors
%
Have there been changes to the zfs implementation between builds 29 and
49 that a pool upgrade would trigger to cause periodic disk activity?
The noise from my disks is slowly driving me crazy (it''s like Chinese
water torture...).
-- Glenn Skinner
Glenn Skinner wrote:> About a month ago, I upgraded my workstation to Nevada build 49. When > I ran "zpool status" on my existing zfs pool (a couple mirrored > partitions, one from each of the system''s two disks), I discovered that > the existing on-disk format was at version 1 and that I could upgrade > to version 3. I chose to defer doing so, until I was sure that I would > no longer boot back into the partition holding my previous root (from > build 29). > > Last week, I decided it was time to take the plunge and invoked > % zpool upgrade zfs > The upgrade completed successfully, with no error messages. > > But since that upgrade I''ve been getting unremitting bursts of disk > i/o. When there''s no activity on the system, they occur at a frequency > of 1/minute. (With activity, the frequency is higher. From listening > to my disks, there seems to be considerably more i/o traffic than > before the upgrade.) Here''s iostat info from when the system is > essentially idle: > > % iostat 20 > tty sd0 sd1 sd3 nfs1 cpu > tin tout kps tps serv kps tps serv kps tps serv kps tps serv us sy wt id > 0 3 273 29 120 280 29 93 0 0 0 0 0 9 26 6 0 69 > 0 12 11 5 20 11 5 17 0 0 0 0 0 0 2 1 0 97 > 0 4 0 0 0 1 1 20 0 0 0 0 0 0 2 1 0 97 > 0 4 2 0 3 1 0 6 0 0 0 0 0 0 2 1 0 97 > 0 4 11 5 20 11 5 18 0 0 0 0 0 0 1 1 0 98 > 0 4 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 98 > 0 4 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 98 > 0 4 11 5 18 11 5 17 0 0 0 0 0 0 1 1 0 98 > 0 4 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 98 > 0 4 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 98 > 0 4 11 5 19 11 5 17 0 0 0 0 0 0 1 1 0 98 > ^C% > > This periodic activity did not occur before I upgraded my zfs pool, and > I can''t think of any other change that might account for it. The pool > is completely healthy: > > % zpool status zfs > pool: zfs > state: ONLINE > scrub: scrub completed with 0 errors on Fri Jan 12 15:34:05 2007 > config: > > NAME STATE READ WRITE CKSUM > zfs ONLINE 0 0 0 > mirror ONLINE 0 0 0 > c0t1d0s6 ONLINE 0 0 0 > c0t0d0s6 ONLINE 0 0 0 > > errors: No known data errors > % > > Have there been changes to the zfs implementation between builds 29 and > 49 that a pool upgrade would trigger to cause periodic disk activity? > The noise from my disks is slowly driving me crazy (it''s like Chinese > water torture...). >Version 2 of ZFS ( http://opensolaris.org/os/community/zfs/version/2/ ) added ditto blocks. That means meta-data is replicated either 2x or 3x on disk. The copies are spread across multiple disks (generally in the dynamically striped case), or across a single disk/partition (generally in the non-striped case). It could be that you''re hearing the disk moving to update the ditto blocks. Though what''s really causing I/O to the devices? Is the traffic from UFS or ZFS? Is it from an application or the kernel? eric
Date: Fri, 19 Jan 2007 16:26:15 -0800
From: eric kustarz <eric.kustarz at sun.com>
Subject: Re: [zfs-discuss] periodic disk i/o upon pool upgrade
[description of my configuration and problem elided]
Version 2 of ZFS (
http://opensolaris.org/os/community/zfs/version/2/ ) added ditto
blocks. That means meta-data is replicated either 2x or 3x on
disk. The copies are spread across multiple disks (generally in
the dynamically striped case), or across a single disk/partition
(generally in the non-striped case).
It could be that you''re hearing the disk moving to update the ditto
blocks.
Though what''s really causing I/O to the devices? Is the traffic
from UFS or ZFS? Is it from an application or the kernel?
I don''t know, although I strongly suspect that it''s ZFS-
rather than
UFS-related, since it became noticeable only after upgrading my ZFS
pool. If you could suggest a DTrace script to run, I''d be happy to do
so.
-- Glenn