Ronny Egner
2010-Feb-26 13:46 UTC
[zfs-discuss] ZFS Storage system with 72 GB memory constantly has 11 GB free memory
Dear All, our storage system running opensolaris b133 + ZFS has a lot of memory for caching. 72 GB total. While testing we observed free memory never falls below 11 GB. Even if we create a ram disk free memory drops below 11 GB but will be 11 GB shortly after (i assume ARC cache is shrunken in this context). As far as i know ZFS is designed to use all memory except 1 GB for caching.... Thanks in advance -- This message posted from opensolaris.org
Thomas Burgess
2010-Feb-26 13:49 UTC
[zfs-discuss] ZFS Storage system with 72 GB memory constantly has 11 GB free memory
i thouhgt it was designed to use 2/3''s of the available memory On Fri, Feb 26, 2010 at 8:46 AM, Ronny Egner <ronnyegner at gmx.de> wrote:> Dear All, > > our storage system running opensolaris b133 + ZFS has a lot of memory for > caching. 72 GB total. While testing we observed free memory never falls > below 11 GB. > > Even if we create a ram disk free memory drops below 11 GB but will be 11 > GB shortly after (i assume ARC cache is shrunken in this context). > > As far as i know ZFS is designed to use all memory except 1 GB for > caching.... > > > > Thanks in advance > -- > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100226/7b580085/attachment.html>
Thomas Burgess
2010-Feb-26 13:51 UTC
[zfs-discuss] ZFS Storage system with 72 GB memory constantly has 11 GB free memory
errr, i mean 3/4...i know it''s some fraction anyways On Fri, Feb 26, 2010 at 8:49 AM, Thomas Burgess <wonslung at gmail.com> wrote:> i thouhgt it was designed to use 2/3''s of the available memory > > > > On Fri, Feb 26, 2010 at 8:46 AM, Ronny Egner <ronnyegner at gmx.de> wrote: > >> Dear All, >> >> our storage system running opensolaris b133 + ZFS has a lot of memory for >> caching. 72 GB total. While testing we observed free memory never falls >> below 11 GB. >> >> Even if we create a ram disk free memory drops below 11 GB but will be 11 >> GB shortly after (i assume ARC cache is shrunken in this context). >> >> As far as i know ZFS is designed to use all memory except 1 GB for >> caching.... >> >> >> >> Thanks in advance >> -- >> This message posted from opensolaris.org >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100226/6ed00e4d/attachment.html>
Robert Milkowski
2010-Feb-26 14:30 UTC
[zfs-discuss] ZFS Storage system with 72 GB memory constantly has 11 GB free memory
on 64bit platforms it is MAX(3/4 of memory, memory - 1GB) by default. so for a system with 72GB it should be MAX(54GB, 71GB) which is 71GB. On 26/02/2010 13:51, Thomas Burgess wrote:> errr, i mean 3/4...i know it''s some fraction anyways > > > On Fri, Feb 26, 2010 at 8:49 AM, Thomas Burgess <wonslung at gmail.com > <mailto:wonslung at gmail.com>> wrote: > > i thouhgt it was designed to use 2/3''s of the available memory > > > > On Fri, Feb 26, 2010 at 8:46 AM, Ronny Egner <ronnyegner at gmx.de > <mailto:ronnyegner at gmx.de>> wrote: > > Dear All, > > our storage system running opensolaris b133 + ZFS has a lot of > memory for caching. 72 GB total. While testing we observed > free memory never falls below 11 GB. > > Even if we create a ram disk free memory drops below 11 GB but > will be 11 GB shortly after (i assume ARC cache is shrunken in > this context). > > As far as i know ZFS is designed to use all memory except 1 GB > for caching.... > > > > Thanks in advance > -- > This message posted from opensolaris.org <http://opensolaris.org> > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org <mailto:zfs-discuss at opensolaris.org> > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100226/255cace0/attachment.html>
Robert Milkowski
2010-Feb-26 14:31 UTC
[zfs-discuss] ZFS Storage system with 72 GB memory constantly has 11 GB free memory
On 26/02/2010 13:46, Ronny Egner wrote:> Dear All, > > our storage system running opensolaris b133 + ZFS has a lot of memory for caching. 72 GB total. While testing we observed free memory never falls below 11 GB. > > Even if we create a ram disk free memory drops below 11 GB but will be 11 GB shortly after (i assume ARC cache is shrunken in this context). > > As far as i know ZFS is designed to use all memory except 1 GB for caching.... > > > > Thanks in advance >can you send output of: mdb -k ::zfs_params ::arc ::memstat ? -- Robert Milkowski http://milek.blogspot.com
Tomas Ögren
2010-Feb-26 14:44 UTC
[zfs-discuss] ZFS Storage system with 72 GB memory constantly has 11 GB free memory
On 26 February, 2010 - Ronny Egner sent me these 0,6K bytes:> Dear All, > > our storage system running opensolaris b133 + ZFS has a lot of memory for caching. 72 GB total. While testing we observed free memory never falls below 11 GB. > > Even if we create a ram disk free memory drops below 11 GB but will be 11 GB shortly after (i assume ARC cache is shrunken in this context). > > As far as i know ZFS is designed to use all memory except 1 GB for caching....http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/fs/zfs/arc.c#arc_init http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/fs/zfs/arc.c#arc_reclaim_needed So you have a max limit which it won''t try to go past, but also a "keep this much free for the rest of the system". Both are a bit too protective for a pure ZFS/NFS server in my opinion (but can be tuned). You can check most variables with f.ex: echo freemem/D | mdb -k On one server here, I have in /etc/system: * http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Limiting_the_ARC_Cache * about 7.8*1024*1024*1024, must be < physmem*pagesize (2062222*4096=8446861312 right now) set zfs:zfs_arc_max = 8350000000 set zfs:zfs_arc_meta_limit = 7000000000 * some tuning set ncsize = 500000 set nfs:nrnode = 50000 And I''ve done runtime modifications to swapfs_minfree to force usage of another chunk of memory. /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se
Richard Elling
2010-Feb-26 15:12 UTC
[zfs-discuss] ZFS Storage system with 72 GB memory constantly has 11 GB free memory
On Feb 26, 2010, at 5:46 AM, Ronny Egner wrote:> Dear All, > > our storage system running opensolaris b133 + ZFS has a lot of memory for caching. 72 GB total. While testing we observed free memory never falls below 11 GB. > > Even if we create a ram disk free memory drops below 11 GB but will be 11 GB shortly after (i assume ARC cache is shrunken in this context). > > As far as i know ZFS is designed to use all memory except 1 GB for caching....In arcstat (or kstats for arc) the "c" value is the target value for "arcsz" which is the current size. http://www.solarisinternals.com/wiki/index.php/Arcstat You can track this over time to see how it reacts. However, if there is a decrease in "c," the cause is not seen from the ZFS perspective and you will need to look for other sources of memory demand. -- richard ZFS storage and performance consulting at http://www.RichardElling.com ZFS training on deduplication, NexentaStor, and NAS performance http://nexenta-atlanta.eventbrite.com (March 16-18, 2010)
Ethan Erchinger
2010-Feb-26 16:16 UTC
[zfs-discuss] ZFS Storage system with 72 GB memoryconstantly has 11 GB free memory
I would probably tune lotsfree down as well. At 72G of ram currently it''s probably reserving around 1.1GB of ram. http://docs.sun.com/app/docs/doc/819-2724/6n50b07bk?a=view Ethan -----Original Message----- From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Tomas ?gren Sent: Friday, February 26, 2010 6:45 AM To: Ronny Egner Cc: zfs-discuss at opensolaris.org Subject: Re: [zfs-discuss] ZFS Storage system with 72 GB memoryconstantly has 11 GB free memory On 26 February, 2010 - Ronny Egner sent me these 0,6K bytes:> Dear All, > > our storage system running opensolaris b133 + ZFS has a lot of memory for caching. 72 GB total. While testing we observed free memory never falls below 11 GB. > > Even if we create a ram disk free memory drops below 11 GB but will be 11 GB shortly after (i assume ARC cache is shrunken in this context). > > As far as i know ZFS is designed to use all memory except 1 GB for caching....http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/fs/zfs/arc.c#arc_init http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/fs/zfs/arc.c#arc_reclaim_needed So you have a max limit which it won''t try to go past, but also a "keep this much free for the rest of the system". Both are a bit too protective for a pure ZFS/NFS server in my opinion (but can be tuned). You can check most variables with f.ex: echo freemem/D | mdb -k On one server here, I have in /etc/system: * http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Limiting_the_ARC_Cache * about 7.8*1024*1024*1024, must be < physmem*pagesize (2062222*4096=8446861312 right now) set zfs:zfs_arc_max = 8350000000 set zfs:zfs_arc_meta_limit = 7000000000 * some tuning set ncsize = 500000 set nfs:nrnode = 50000 And I''ve done runtime modifications to swapfs_minfree to force usage of another chunk of memory. /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Ronny Egner
2010-Feb-26 18:11 UTC
[zfs-discuss] ZFS Storage system with 72 GB memory constantly has 11 GB free memory
Hi, please find below the requested information: root at openstorage:~# mdb -k Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp rootnex scsi_vhci zfs sockfs ip hook neti sctp arp usba uhci fctl stmf md lofs idm random mpt sd nfs crypto fcp fcip cpc smbsrv ufs logindmux ptm sppp nsmb ii nsctl rdc sv sdbc ]> ::zfs_paramsarc_reduce_dnlc_percent = 0x3 zfs_arc_max = 0x0 zfs_arc_min = 0x0 arc_shrink_shift = 0x5 zfs_mdcomp_disable = 0x0 zfs_prefetch_disable = 0x0 zfetch_max_streams = 0x8 zfetch_min_sec_reap = 0x2 zfetch_block_cap = 0x100 zfetch_array_rd_sz = 0x100000 zfs_default_bs = 0x9 zfs_default_ibs = 0xe metaslab_aliquot = 0x80000 mdb: variable reference_tracking_enable not found: unknown symbol name mdb: variable reference_history not found: unknown symbol name spa_max_replication_override = 0x3 spa_mode_global = 0x3 zfs_flags = 0x0 mdb: variable zfs_txg_synctime not found: unknown symbol name zfs_txg_timeout = 0x1e zfs_write_limit_min = 0x2000000 zfs_write_limit_max = 0x23fde5800 zfs_write_limit_shift = 0x3 zfs_write_limit_override = 0x0 zfs_no_write_throttle = 0x0> ::archits = 682809234 misses = 41519142 demand_data_hits = 26047450 demand_data_misses = 17440267 demand_metadata_hits = 636130758 demand_metadata_misses = 15436051 prefetch_data_hits = 10328015 prefetch_data_misses = 8549656 prefetch_metadata_hits = 10303011 prefetch_metadata_misses = 93168 mru_hits = 15961928 mru_ghost_hits = 287507 mfu_hits = 655313464 mfu_ghost_hits = 14603118 deleted = 43957777 recycle_miss = 448696 mutex_miss = 572393 evict_skip = 99942 evict_l2_cached = 0 evict_l2_eligible = 421991499264 evict_l2_ineligible = 30080494080 hash_elements = 5756890 hash_elements_max = 10234471 hash_collisions = 51949950 hash_chains = 1583031 hash_chain_max = 19 p = 32573 MB c = 42754 MB c_min = 9085 MB c_max = 72687 MB size = 42754 MB hdr_size = 1105922976 data_size = 43097086464 other_size = 627925600 l2_hits = 0 l2_misses = 0 l2_feeds = 0 l2_rw_clash = 0 l2_read_bytes = 0 l2_write_bytes = 0 l2_writes_sent = 0 l2_writes_done = 0 l2_writes_error = 0 l2_writes_hdr_miss = 0 l2_evict_lock_retry = 0 l2_evict_reading = 0 l2_free_on_write = 0 l2_abort_lowmem = 0 l2_cksum_bad = 0 l2_io_error = 0 l2_size = 0 l2_hdr_size = 0 memory_throttle_count = 0 arc_no_grow = 1 arc_tempreserve = 0 MB arc_meta_used = 9977 MB arc_meta_limit = 18171 MB arc_meta_max = 16326 MB> ::memstatPage Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 7676041 29984 41% ZFS File Data 8150818 31839 43% Anon 31940 124 0% Exec and libs 1605 6 0% Page cache 5726 22 0% Free (cachelist) 429684 1678 2% Free (freelist) 2574246 10055 14% Total 18870060 73711 Physical 18870059 73711 -- This message posted from opensolaris.org
Richard Elling
2010-Feb-26 19:04 UTC
[zfs-discuss] ZFS Storage system with 72 GB memory constantly has 11 GB free memory
explanation below... On Feb 26, 2010, at 10:11 AM, Ronny Egner wrote:> Hi, > > please find below the requested information: > > root at openstorage:~# mdb -k > Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp rootnex scsi_vhci zfs sockfs ip hook neti sctp arp usba uhci fctl stmf md lofs idm random mpt sd nfs crypto fcp fcip cpc smbsrv ufs logindmux ptm sppp nsmb ii nsctl rdc sv sdbc ] > >> ::zfs_params > arc_reduce_dnlc_percent = 0x3 > zfs_arc_max = 0x0 > zfs_arc_min = 0x0 > arc_shrink_shift = 0x5 > zfs_mdcomp_disable = 0x0 > zfs_prefetch_disable = 0x0 > zfetch_max_streams = 0x8 > zfetch_min_sec_reap = 0x2 > zfetch_block_cap = 0x100 > zfetch_array_rd_sz = 0x100000 > zfs_default_bs = 0x9 > zfs_default_ibs = 0xe > metaslab_aliquot = 0x80000 > mdb: variable reference_tracking_enable not found: unknown symbol name > mdb: variable reference_history not found: unknown symbol name > spa_max_replication_override = 0x3 > spa_mode_global = 0x3 > zfs_flags = 0x0 > mdb: variable zfs_txg_synctime not found: unknown symbol name > zfs_txg_timeout = 0x1e > zfs_write_limit_min = 0x2000000 > zfs_write_limit_max = 0x23fde5800 > zfs_write_limit_shift = 0x3 > zfs_write_limit_override = 0x0 > zfs_no_write_throttle = 0x0 >Note: "kstat -n arcstats" allows you to see these variables without the need to be root or use mdb.>> ::arc > hits = 682809234 > misses = 41519142 > demand_data_hits = 26047450 > demand_data_misses = 17440267 > demand_metadata_hits = 636130758 > demand_metadata_misses = 15436051 > prefetch_data_hits = 10328015 > prefetch_data_misses = 8549656 > prefetch_metadata_hits = 10303011 > prefetch_metadata_misses = 93168 > mru_hits = 15961928 > mru_ghost_hits = 287507 > mfu_hits = 655313464 > mfu_ghost_hits = 14603118 > deleted = 43957777 > recycle_miss = 448696 > mutex_miss = 572393 > evict_skip = 99942 > evict_l2_cached = 0 > evict_l2_eligible = 421991499264 > evict_l2_ineligible = 30080494080 > hash_elements = 5756890 > hash_elements_max = 10234471 > hash_collisions = 51949950 > hash_chains = 1583031 > hash_chain_max = 19 > p = 32573 MB > c = 42754 MBtarget ARC size> c_min = 9085 MB > c_max = 72687 MBupper limit to the target ARC size> size = 42754 MBcurrent size So you can see that the ARC will allow itself to grow to 72GB - 1GB. However, the current size is the same as the target size and far less than target max, which can indicate...> hdr_size = 1105922976 > data_size = 43097086464 > other_size = 627925600 > l2_hits = 0 > l2_misses = 0 > l2_feeds = 0 > l2_rw_clash = 0 > l2_read_bytes = 0 > l2_write_bytes = 0 > l2_writes_sent = 0 > l2_writes_done = 0 > l2_writes_error = 0 > l2_writes_hdr_miss = 0 > l2_evict_lock_retry = 0 > l2_evict_reading = 0 > l2_free_on_write = 0 > l2_abort_lowmem = 0 > l2_cksum_bad = 0 > l2_io_error = 0 > l2_size = 0 > l2_hdr_size = 0 > memory_throttle_count = 0 > arc_no_grow = 1flag to indicate whether the ARC will try to grow. When it is set to 1, the ARC won''t try to grow for another 60 seconds. This indicates that the ARC was recently asked to reclaim space. Three conditions can cause this for x86: 1. pageout scanner is running (a small margin applies here) 2. swapfs does not have enough space so that anonymous reservations can succeed. This is calculated from as: swapfs_minfree + swapfs_reserv + desfree on one of my machines with 2 GBytes of RAM, this limit is 73,437 pages (300,797,952 bytes). hint: "echo swapfs_minfree/D" | mdb -k 3. [x86 only] kernel heap space is more than 75% allocated. IIRC, this is more of a problem for 32-bit systems. Unless I''m mistaken, you can see the kernel heap arena size use ratio by looking at the "mem_inuse" to "mem_total" ratio via "kstat -n heap" -- richard> arc_tempreserve = 0 MB > arc_meta_used = 9977 MB > arc_meta_limit = 18171 MB > arc_meta_max = 16326 MB > > >> ::memstat > Page Summary Pages MB %Tot > ------------ ---------------- ---------------- ---- > Kernel 7676041 29984 41% > ZFS File Data 8150818 31839 43% > Anon 31940 124 0% > Exec and libs 1605 6 0% > Page cache 5726 22 0% > Free (cachelist) 429684 1678 2% > Free (freelist) 2574246 10055 14% > > Total 18870060 73711 > Physical 18870059 73711 > -- > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discussZFS storage and performance consulting at http://www.RichardElling.com ZFS training on deduplication, NexentaStor, and NAS performance http://nexenta-atlanta.eventbrite.com (March 16-18, 2010)
Ronny Egner
2010-Feb-26 20:20 UTC
[zfs-discuss] ZFS Storage system with 72 GB memory constantly has 11 GB free memory
Hi Richard, root at openstorage:~# echo "swapfs_minfree/D" | mdb -k swapfs_minfree: swapfs_minfree: 2358757 2358757 pages * 4 KB = 9435028 Kb = 9.2 GB So there is my memory :-) If i read the documentation correctly this parameter has to be set in /etc/system? Ronny -- This message posted from opensolaris.org
Richard Elling
2010-Feb-26 22:58 UTC
[zfs-discuss] ZFS Storage system with 72 GB memory constantly has 11 GB free memory
On Feb 26, 2010, at 12:20 PM, Ronny Egner wrote:> Hi Richard, > > root at openstorage:~# echo "swapfs_minfree/D" | mdb -k > swapfs_minfree: > swapfs_minfree: 2358757 > > > 2358757 pages * 4 KB = 9435028 Kb = 9.2 GB > > So there is my memory :-)I believe so.> > If i read the documentation correctly this parameter has to be set in /etc/system?There is an interesting (to me :-) discussion about this in http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=4469865 The way I see this is that the rule of 7/8 memory or memory - 1GB is, for all practical purposes, just 7/8 of memory because the ARC will begin reclaiming as it crosses the 7/8 threshold. Obviously, the box needs to have more than 8GB before this is noticeable. I think it is quite reasonable to set this to a lower value for a storage server with lots of memory because you don''t expect to have a sudden need for large quantities of swap. See also http://docs.sun.com/app/docs/doc/819-2724/chapter2-125?a=view -- richard ZFS storage and performance consulting at http://www.RichardElling.com ZFS training on deduplication, NexentaStor, and NAS performance http://nexenta-atlanta.eventbrite.com (March 16-18, 2010)