thr3ads.net - zfs discuss - [zfs-discuss] ZFS slows down over a couple of days [Jan 2011]

If this information is useful, please help other people find it:
Share via:

Stephan Budach

2011-Jan-12 15:21 UTC

[zfs-discuss] ZFS slows down over a couple of days

Hi all,

I have exchanged my Dell R610 in favor of a Sun Fire 4170 M2 which has 
32 GB RAM installed. I am running Sol11Expr on this host and I use it to 
primarily serve Netatalk AFP shares. From day one, I have noticed that 
the amount of free RAM decereased and along with that  decrease the 
overall performance of ZFS decreased as well.

Now, since I am still quite a Solaris newbie, I seem to cannot track 
where the heck all the memory has gone and why ZFS performs so poorly 
after an uptime of only 5 days.
I can reboot Solaris, which I did for testing, and that would bring back 
the performance to reasonable levels, but otherwiese I am quite at my 
witts end.
To give some numbers: the ZFS performance decreases down to 1/10th of 
the initial throughput, either read or write.

Anybody having some tips up their sleeves, where I should start looking 
for the missing memory?

Cheers,
budy
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110112/ea8d532f/attachment.html>

Jeff Savit

2011-Jan-12 15:32 UTC

head link

[zfs-discuss] ZFS slows down over a couple of days

Stephan,

There are a bunch of tools you can use, mostly provided with Solaris 11 
Express, plus arcstat, arc_summary that are available as downloads.  The 
latter tools will tell you the size and state of ARC, which may be 
specific to your issues since you cite memory.   For the list, could you 
describe the ZFS pool configuration (zpool status), and summarize output 
from vmstat, iostat, and zpool iostat?  Also, it might be helpful to 
issue ''prstat -s rss'' to see if any process is growing its
resident
memory size.  An excellet source of information is the "ZFS evil tuning 
guide" (just Google those words), which has a wealth of information.

I hope that helps (for a start at least)
   Jeff



On 01/12/11 08:21 AM, Stephan Budach wrote:> Hi all,
>
> I have exchanged my Dell R610 in favor of a Sun Fire 4170 M2 which has 
> 32 GB RAM installed. I am running Sol11Expr on this host and I use it 
> to primarily serve Netatalk AFP shares. From day one, I have noticed 
> that the amount of free RAM decereased and along with that  decrease 
> the overall performance of ZFS decreased as well.
>
> Now, since I am still quite a Solaris newbie, I seem to cannot track 
> where the heck all the memory has gone and why ZFS performs so poorly 
> after an uptime of only 5 days.
> I can reboot Solaris, which I did for testing, and that would bring 
> back the performance to reasonable levels, but otherwiese I am quite 
> at my witts end.
> To give some numbers: the ZFS performance decreases down to 1/10th of 
> the initial throughput, either read or write.
>
> Anybody having some tips up their sleeves, where I should start 
> looking for the missing memory?
>
> Cheers,
> budy
-- 


*Jeff Savit* | Principal Sales Consultant
Phone: 602.824.6275 | Email: jeff.savit at oracle.com | Blog: 
http://blogs.sun.com/jsavit
Oracle North America Commercial Hardware
Operating Environments & Infrastructure S/W Pillar
2355 E Camelback Rd | Phoenix, AZ 85016



-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110112/8cdab402/attachment-0001.html>

Stephan Budach

2011-Jan-12 16:38 UTC

head link

[zfs-discuss] ZFS slows down over a couple of days

Am 12.01.11 16:32, schrieb Jeff Savit:> Stephan,
>
> There are a bunch of tools you can use, mostly provided with Solaris 
> 11 Express, plus arcstat, arc_summary that are available as 
> downloads.  The latter tools will tell you the size and state of ARC, 
> which may be specific to your issues since you cite memory.   For the 
> list, could you describe the ZFS pool configuration (zpool status), 
> and summarize output from vmstat, iostat, and zpool iostat?  Also, it 
> might be helpful to issue ''prstat -s rss'' to see if any
process is
> growing its resident memory size.  An excellet source of information 
> is the "ZFS evil tuning guide" (just Google those words), which
has a
> wealth of information.
>
> I hope that helps (for a start at least)
>   Jeff
>
>
>
> On 01/12/11 08:21 AM, Stephan Budach wrote:
>> Hi all,
>>
>> I have exchanged my Dell R610 in favor of a Sun Fire 4170 M2 which 
>> has 32 GB RAM installed. I am running Sol11Expr on this host and I 
>> use it to primarily serve Netatalk AFP shares. From day one, I have 
>> noticed that the amount of free RAM decereased and along with that  
>> decrease the overall performance of ZFS decreased as well.
>>
>> Now, since I am still quite a Solaris newbie, I seem to cannot track 
>> where the heck all the memory has gone and why ZFS performs so poorly 
>> after an uptime of only 5 days.
>> I can reboot Solaris, which I did for testing, and that would bring 
>> back the performance to reasonable levels, but otherwiese I am quite 
>> at my witts end.
>> To give some numbers: the ZFS performance decreases down to 1/10th of 
>> the initial throughput, either read or write.
>>
>> Anybody having some tips up their sleeves, where I should start 
>> looking for the missing memory?
>>
>> Cheers,
>> budy
>Sure - here we go. First of all, the zpool configuration:

zpool status -v
   pool: obelixData
  state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
     still be used, but some features are unavailable.
action: Upgrade the pool using ''zpool upgrade''.  Once this is
done, the
     pool will no longer be accessible on older software versions.
  scan: scrub repaired 0 in 15h29m with 0 errors on Mon Nov 15 21:42:52 2010
config:

     NAME                     STATE     READ WRITE CKSUM
     obelixData               ONLINE       0     0     0
       c9t210000D023038FA8d0  ONLINE       0     0     0
       c9t210000D02305FF42d0  ONLINE       0     0     0

errors: No known data errors

This pool consists of two FC LUNS which are exported from two FC RAIDs 
(no comments on that one, please I am still working on the transision to 
another zpool config! ;) )

Next up are arcstat.pl and arc_summary.pl:

perl /usr/local/de.jvm.scripts/arcstat.pl
time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c
17:13:33     0     0      0     0    0     0    0     0    0    15G   16G
17:13:34    71     0      0     0    0     0    0     0    0    15G   16G
17:13:35     3     0      0     0    0     0    0     0    0    15G   16G
17:13:36   30K     0      0     0    0     0    0     0    0    15G   16G
17:13:37   13K     0      0     0    0     0    0     0    0    15G   16G
17:13:38    72     0      0     0    0     0    0     0    0    15G   16G
17:13:39    12     0      0     0    0     0    0     0    0    15G   16G
17:13:40    45     0      0     0    0     0    0     0    0    15G   16G
17:13:41    57     0      0     0    0     0    0     0    0    15G   16G
<State Changed>
17:13:42  1.3K     8      0     8    0     0    0     6    0    15G   16G
17:13:43    45     0      0     0    0     0    0     0    0    15G   16G
17:13:44  1.5K    15      1    13    0     2   50     4    0    15G   16G
17:13:45   122     0      0     0    0     0    0     0    0    15G   16G
17:13:46    74     0      0     0    0     0    0     0    0    15G   16G
17:13:47    88     0      0     0    0     0    0     0    0    15G   16G
17:13:48   19K    67      0    25    0    42    4    24    0    16G   16G
17:13:49   24K    31      0     0    0    31    9     0    0    15G   16G
17:13:50    41     0      0     0    0     0    0     0    0    15G   16G

perl /usr/local/de.jvm.scripts/arc_summary.pl
System Memory:
      Physical RAM:     32751 MB
      Free Memory :     5615 MB
      LotsFree:     511 MB

ZFS Tunables (/etc/system):
      set zfs:zfs_arc_max = 17179869184

ARC Size:
      Current Size:             16383 MB (arcsize)
      Target Size (Adaptive):   16384 MB (c)
      Min Size (Hard Limit):    2048 MB (zfs_arc_min)
      Max Size (Hard Limit):    16384 MB (zfs_arc_max)

ARC Size Breakdown:
      Most Recently Used Cache Size:      73%     12015 MB (p)
      Most Frequently Used Cache Size:      26%     4368 MB (c-p)

ARC Efficency:
      Cache Access Total:             300030668
      Cache Hit Ratio:      92%     277102547       [Defined State for 
buffer]
      Cache Miss Ratio:      7%     22928121       [Undefined State for 
Buffer]
      REAL Hit Ratio:       84%     253621864       [MRU/MFU Hits Only]

      Data Demand   Efficiency:    98%
      Data Prefetch Efficiency:    26%

     CACHE HITS BY CACHE LIST:
       Anon:                        4%      12706439                [ 
New Customer, First Cache Hit ]
       Most Recently Used:         12%      35919656 (mru)          [ 
Return Customer ]
       Most Frequently Used:       78%      217702208 (mfu)          [ 
Frequent Customer ]
       Most Recently Used Ghost:    0%      2469558 (mru_ghost)    [ 
Return Customer Evicted, Now Back ]
       Most Frequently Used Ghost:  2%      8304686 (mfu_ghost)    [ 
Frequent Customer Evicted, Now Back ]
     CACHE HITS BY DATA TYPE:
       Demand Data:                38%      105648747
       Prefetch Data:               1%      3607719
       Demand Metadata:            53%      147826874
       Prefetch Metadata:           7%      20019207
     CACHE MISSES BY DATA TYPE:
       Demand Data:                 9%      2130351
       Prefetch Data:              43%      9907868
       Demand Metadata:            23%      5396413
       Prefetch Metadata:          23%      5493489
---------------------------------------------

So, these show me, that ZFS ARC actually shouldn''t have grown over
16GB,
right?

Now for vmstat/iostat/prstat:

vmstat 10
  kthr      memory            page            disk          faults      cpu
  r b w   swap  free  re  mf pi po fr de sr s0 s1 s2 s3   in   sy   cs 
us sy id
  0 0 0 22210512 9676332 29 215 0 0 0  0  1 -0  8  8 40 6460 13462 5409 
2  1 97
  0 1 0 18193100 5752352 32 305 0 0 0  0  0  1 15 15 19 3245 29489 1786 
0  1 98
  0 1 0 18193112 5752084 11 223 0 0 0  0  0  0  5  5 19 3628 30364 2219 
0  2 97
  0 1 0 18193132 5752568 1 139 0 0  0  0  0  0 21 20 24 4125 31415 2432 
0  2 98
  0 1 0 18193140 5752592 25 89 0 0  0  0  0  0  0  0 19 3070 23238 1537 
0  1 99
  0 1 0 18193204 5752316 10 167 0 0 0  0  0  0  8  8 29 4117 14217 2810 
0  2 98


As you can see, the free memory shrinked from 32 GB to approx 5 GB. 
Substracting the ZFS ARC from the 32 GB, that''d leave another 16 GB for
Solaris and its crops. So actually there have been 11 GB consumed, which 
I can''t pinpoint using prstat. Although 5 GB od freem RAM should be 
still enough, the performance drops as soon as the free memory falls 
below 10 GB, which is pretty much the case after 2 work days.

iostat 10
    tty        sd0           sd1           sd2           sd3            cpu
  tin tout kps tps serv  kps tps serv  kps tps serv  kps tps serv   us 
sy wt id
    0   24   0   0    0   61   8    3   61   8    3  1107  40    6    2  
1  0 97
    0   97   0   0    0  160  16    3  160  16    3  170  15    5    0  
2  0 98
    0  175   0   0    0  168  20    3  168  20    3  105  15    3    0  
2  0 98
    0  111   0   0    0    0   0    0    0   0    0   90  14    4    0  
1  0 99
    0   62   0   0    0  146  13    3  146  13    3  422  31    5    0  
2  0 98

iostat seems to show a quite idle system...

Last, but not least: zpool iostat barely sees read or write throughputs 
above 35 MB/s.

Btw, this is regardless whether I use netatlk or smb for sharing.

Thanks,
budy
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110112/9b326e83/attachment.html>

2011-Jan-12 17:49 UTC

head link

[zfs-discuss] ZFS slows down over a couple of days

You may need to adjust zfs_arc_max in /etc/system to avoid memory contention


http://www.thezonemanager.com/2009/03/filesystem-cache-optimization.htm

Suresh
-- 
This message posted from opensolaris.org

Stephan Budach

2011-Jan-12 18:16 UTC

head link

[zfs-discuss] ZFS slows down over a couple of days

Am 12.01.11 18:49, schrieb SR:> You may need to adjust zfs_arc_max in /etc/system to avoid memory
contention
>
>
> http://www.thezonemanager.com/2009/03/filesystem-cache-optimization.htm
>
> SureshI though I had that done through this in /etc/system:

set zfs:zfs_arc_max = 17179869184


I do also think that arc_summary.pl showed exactly that...

Cheers,
budy


-- 
Stephan Budach
Jung von Matt/it-services GmbH
Glash?ttenstra?e 79
20357 Hamburg

Tel: +49 40-4321-1353
Fax: +49 40-4321-1114
E-Mail: stephan.budach at jvm.de
Internet: http://www.jvm.com

Gesch?ftsf?hrer: Ulrich Pallas, Frank Wilhelm
AG HH HRB 98380

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110112/1a996ef9/attachment-0001.html>

Marion Hakanson

2011-Jan-12 19:23 UTC

head link

[zfs-discuss] ZFS slows down over a couple of days

Stephan,

The "vmstat" shows you are not actually short of memory;  The
"pi" and "po"
columns are zero, so the system is not having to do any paging, and it seems
unlike the system is slow directly because of RAM shortage.  With the ARC,
it''s not unusual for vmstat to show little free memory, but the system
will
give up that RAM when an application asks for it.  You can tell if this is
happening a lot by:
	echo "::arc" | mdb -k | grep throttle

If the value of "memory_throttle_count" is large, that will indicate
that
apps are often asking the kernel to give up ARC memory.

Also, as you said, the "iostat" figures look idle.  You can tell more
using "iostat -xn 1", which will give service times & percent-busy
figures for the actual devices.

It could be that something about the networking involved is what is
actually slow.  You could find out if it''s a local bottleneck by trying
some simple I/O tests on the server itself, maybe:
	dd if=/dev/zero of=/file/in/zpool bs=1024k
and watching what iostat shows, etc.

Another test is to try a network-only test, maybe using "ttcp" between
the server and a client.  This could tell you if it''s network or
storage
that''s causing the slow-down.  If you don''t have
"ttcp", something silly
like, on a client running:
	dd if=/dev/zero bs=1024k | ssh -c blowfish server "dd of=/dev/null
bs=1024k"

You can watch network throughput on the server using:
	dladm show-link -s -i 1

Regards,

Marion

Stephan Budach

2011-Jan-13 08:01 UTC

head link

[zfs-discuss] ZFS slows down over a couple of days

Hi all,

thanks a lot for your suggestions. I have checked all of them and 
neither the network itself nor any other check indicated any problem.

Alas, I think I know what is going on? ehh? my current zpool has two 
vdevs that are actually not even sized, as shown by zpool iostat -v:

zpool iostat -v obelixData 5
capacity operations bandwidth
pool alloc free read write read write
----------------------- ----- ----- ----- ----- ----- -----
obelixData 13,1T 5,84T 36 227 348K 21,5M
c9t210000D023038FA8d0 6,25T 59,3G 21 98 269K 9,25M
c9t210000D02305FF42d0 6,84T 5,78T 15 129 79,2K 12,3M
----------------------- ----- ----- ----- ----- ----- -----


So, the small vdev is actually 99+% full, which is likely to be the root 
cause for this issue. Especially, since RAIDs tend to take tremendous 
performance hits, when they exceed 90% space utilization.

Possibly Parallel Threads

Search for more possibly parallel threads

zfs discuss - Jan 2011 - ZFS slows down over a couple of days

[zfs-discuss] ZFS slows down over a couple of days

[zfs-discuss] ZFS slows down over a couple of days

[zfs-discuss] ZFS slows down over a couple of days

[zfs-discuss] ZFS slows down over a couple of days

[zfs-discuss] ZFS slows down over a couple of days

[zfs-discuss] ZFS slows down over a couple of days

[zfs-discuss] ZFS slows down over a couple of days

Possibly Parallel Threads