zfs discuss - Oct 2006 - ?: cyclical kernel/system processing (approx every 4 minutes)

Customer benchmarked an X4600 using UFS on top of VxVM a while back 
and got consistent performance under heavy load. Now they have put the 
system into system test, but in the process moved from UFS/VxVM to 
ZFS. This is 6/06

The are running at approximately 40% idle most of the time, with 10+% 
system time, and 40+% user time. However, every 4 minutes or so, the 
system time goes up to 60%, leaving no idle time. This is under steady 
load, and the customer is concerned that it will not be able to ride 
out peaks higher than this.

Is there any known processing that would be done every 4 minutes last 
10 to 15 seconds that might be introduced by ZFS?

I don''t have sufficient data yet to verify that the I/O is going to 
ZFS volumes, but I do see a pair of disk with

                   extended device statistics
device       r/s    w/s   kr/s   kw/s wait actv  svc_t  %w  %b
sd21         0.1  129.3    8.5 9396.0  0.0  0.9    6.7   0   7
sd22         0.1  138.7    8.5 9872.0  0.0  0.9    6.7   0   7

going to:
                   extended device statistics
device       r/s    w/s   kr/s   kw/s wait actv  svc_t  %w  %b
sd21         0.0   59.0    0.0 1383.4  0.0  0.0    0.8   0   1
sd22         0.0   61.5    0.0 1455.2  0.0  0.1    0.9   0   1

or to:

                   extended device statistics
device       r/s    w/s   kr/s   kw/s wait actv  svc_t  %w  %b
sd21         0.0  180.2    2.1 16513.5  0.0  2.5   14.0   0  11
sd22         0.0  195.2    2.1 17485.1  0.0  3.0   15.5   0  11

Don''t know yet which is ''normal'' and which is not.

Thanks for any insights!
Steffen

Steffen Weiberle wrote:
> Is there any known processing that would be done every 4 minutes last 10 
> to 15 seconds that might be introduced by ZFS?
No.  But you can use tools like dtrace or lockstat -kgIW to determine 
where that additional CPU time is being spent, and then see if ZFS is to 
blame...

--matt