zfs discuss - Jan 2011 - ZFS percent busy vs zpool iostat

Hi,

  I''ve been looking at a performance issue on my system, its FreeBSD
8.1, zpool version 14 (hope Im allowed to ask about FreeBSD on here?!). Anyway I
think its quite a general question. The pool contains a single 2 disk mirror,
using 5k SATA drives.

I am monitoring the percent busy, and the data in/out on the physical disks and
I am seeing close to 100% busy often. Thats ok, these things can happen! ;). The
thing I don''t understand is why during these busy times that zpool
iostat shows very low IO. During a 10 second period the physical disks are close
to 100% busy and show about 5MB/sec IO, during the same period zpool shows
something like this:

mx0         52.9G   691G     75      0   195K      0
mx0         52.9G   691G    178      0   430K      0
mx0         52.9G   691G    200      1   513K  35.9K
mx0         52.9G   691G    121      6   301K   688K
mx0         52.9G   691G    207      0   526K      0
mx0         52.9G   691G    281      0   708K      0
mx0         52.9G   691G    249      0   676K      0
mx0         52.9G   691G    191      0   518K      0
mx0         52.9G   691G    169      0   452K   128K
mx0         52.9G   691G    144      2   363K   383K
mx0         52.9G   691G    140      0   365K      0
mx0         52.9G   691G    151      0   395K   128K


So this is showing mostly reads, but nothing very taxing. Additionally my ARC
cache is not being fully utilised:

ARC Size:
        Current Size:                   28.75%  603.23M (arcsize)
        Target Size: (Adaptive)         30.35%  636.68M (c)
        Min Size (Hard Limit):          12.50%  262.26M (c_min)
        Max Size (High Water):          ~8:1    2098.08M (c_max)

So more ARC isn''t going to help me.

My goal being, what is the best way to improve my system performance, more RAM,
more disks, dedicated LOG or ARC2 disks etc etc. The load appears in zpool
iostat to be mostly read, but my ARC isnt even fully utilised so that to be
seems an odd one.

I know I don''t have high end hardware, and only 2 disks. But
I''d like to understand the apparent disparity between the physical disk
IO and the zpool IO output. That way I can better decide if I should, for
example, add 2x SSDs for ZIL (and L2ARC if it was needed, but seems not to be)
or replace the existing disks and put instead 2x 2 way mirror on slightly better
7k SATA drives.

Anyone able to throw some light on what might be going on under the hood?

thanks! Andy.
-- 
This message posted from opensolaris.org

On Tue, 11 Jan 2011, Andy wrote:
> I am monitoring the percent busy, and the data in/out on the 
> physical disks and I am seeing close to 100% busy often. Thats ok, 
> these things can happen! ;). The thing I don''t understand is why 
> during these busy times that zpool iostat shows very low IO. During 
> a 10 second period the physical disks are close to 100% busy and 
> show about 5MB/sec IO, during the same period zpool shows something 
> like this:
What function is the system performing when it is so busy?
> So this is showing mostly reads, but nothing very taxing. 
> Additionally my ARC cache is not being fully utilised:
>
> ARC Size:
>        Current Size:                   28.75%  603.23M (arcsize)
>        Target Size: (Adaptive)         30.35%  636.68M (c)
>        Min Size (Hard Limit):          12.50%  262.26M (c_min)
>        Max Size (High Water):          ~8:1    2098.08M (c_max)
>
> So more ARC isn''t going to help me.
Wrong conclusion.  I am not sure what the percentages are percentages 
of (total RAM?), but 603MB is a very small ARC.  FreeBSD pre-assigns 
kernel memory for zfs so it is not dynamically shared with the kernel 
as it is with Solaris.  The FreeBSD zfs tuning is quite FreeBSD 
specific.  Even if you have a huge amount of physical RAM, you may 
need to tune FreeBSD to actually use it.  Post a query on 
freebsd-fs at freebsd.org if the FreeBSD zfs tuning pages don''t reveal 
what you need.
> My goal being, what is the best way to improve my system 
> performance, more RAM, more disks, dedicated LOG or ARC2 disks etc 
> etc. The load appears in zpool iostat to be mostly read, but my ARC 
> isnt even fully utilised so that to be seems an odd one.
The ARC is "adaptive" so you should not assume that its objective is 
to try to absorb your hard drive.  It should not want to cache data 
which is rarely accessed.  Regardless,  your ARC size may actually be 
constrained by default FreeBSD kernel tunings.
> I know I don''t have high end hardware, and only 2 disks. But
I''d
> like to understand the apparent disparity between the physical disk 
> IO and the zpool IO output. That way I can better decide if I 
> should, for example, add 2x SSDs for ZIL (and L2ARC if it was 
> needed, but seems not to be) or replace the existing disks and put 
> instead 2x 2 way mirror on slightly better 7k SATA drives.
The type of drives you are using have very poor seek performance. 
Higher RPM drives would surely help.  Stuffing lots more memory in 
your system and adjusting the kernel so that zfs can use a lot more of 
it is likely to help dramatically.  Zfs loves memory.

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

Quoting Bob Friesenhahn <bfriesen at simple.dallas.tx.us>:
>
> What function is the system performing when it is so busy?
The work load of the server is SMTP mail server, with associated spam  
and virus scanning, and serving maildir email via POP3 and IMAP.
>
> Wrong conclusion.  I am not sure what the percentages are  
> percentages of (total RAM?), but 603MB is a very small ARC.  FreeBSD  
> pre-assigns kernel memory for zfs so it is not dynamically shared  
> with the kernel as it is with Solaris.
This is the min, max, and actual size of the ARC. ZFS is free to use  
up to the MAX (2098.08M) if it decides it wants to. Depending on the  
work load on this server it will go up to 2098M (as in Ive seen it get  
to that size on this and other servers), just with its usual daily  
work load it decides to set this to around 600M. I assume it decides  
it''s not worth using any more RAM.
> The ARC is "adaptive" so you should not assume that its objective
is
> to try to absorb your hard drive.  It should not want to cache data  
> which is rarely accessed.  Regardless,  your ARC size may actually  
> be constrained by default FreeBSD kernel tunings.
I guess then that ZFS is weighing up how useful it is to use more than  
600M and deciding that it isnt that useful....? Anyway, Ive just  
forced the Min to 1900M so will see how this goes today.
>
> The type of drives you are using have very poor seek performance.  
> Higher RPM drives would surely help.  Stuffing lots more memory in  
> your system and adjusting the kernel so that zfs can use a lot more  
> of it is likely to help dramatically.  Zfs loves memory.
thanks Bob, and also to Matt for your comments...

Ok, think I have the biggest issue. The drives are 4k sector drives,  
and I wasn''t aware of that. My fault, I should have checked this. Had  
the disks for ages and are sub 1TB so had the idea that they wouldn''t  
be 4k drives...

I will obviously have to address this, either by creating a pool using  
4k aware zfs commands or replacing the disks.

Anyway, thanks to all and to Taemun for getting me to check this...