Tomas Ögren
2007-Jan-03 20:31 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory exhaustion(?) with snv53
On 03 January, 2007 - Richard Elling sent me these 0,5K bytes:> Tomas ?gren wrote: > >df (GNU df) says there are ~850k inodes used, I''d like to keep those in > >memory.. There is currently 1.8TB used on the filesystem.. The > >probability of a cache hit in the user data cache is about 0% and the > >probability that an rsync happens again shortly is about 100%.. > > Also, buy more RAM.Been there, done that.. That''s why the machine has 2GB now.. going over that for a Blade1000 isn''t that cheap..> Sometimes, throwing hardware at a big problem is a good answer :-)Although in this case it''s a known memory leak (will upgrade to snv54 tomorrow which has it fixed), where adding more memory won''t fix the problem.. http://bugs.opensolaris.org/view_bug.do?bug_id=6493923 /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se
Tomas Ögren
2007-Jan-04 22:48 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory exhaustion(?) with snv53
On 03 January, 2007 - Casper.Dik at Sun.COM sent me these 0,5K bytes:> > >Hmmm, so there is lots of evictable cache here (mostly in the MFU > >part of the cache)... could you make your core file available? > >I would like to take a look at it. > > Isn''t this just like: > 6493923 nfsfind on ZFS filesystem quickly depletes memory in a 1GB system > > Which was introduced in b51(or 52) and fixed in snv_54.I''ve upgraded to snv54 now and even forced 500k dnlc entries in memory (ncsize=500000,arc_reduce_dnlc_percent=0).. Let''s see how it copes with that ;) Currently seeing about 96% cache hit rate in name lookups even after just 3h.. it''s usually been around 20% or so when it''s automatically lowered to around 15k entries due to memory pressure.. /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se
Tomas Ögren
2007-Jan-05 03:00 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory exhaustion(?) with snv53
On 04 January, 2007 - Tomas ?gren sent me these 1,0K bytes:> On 03 January, 2007 - Casper.Dik at Sun.COM sent me these 0,5K bytes: > > > > > >Hmmm, so there is lots of evictable cache here (mostly in the MFU > > >part of the cache)... could you make your core file available? > > >I would like to take a look at it. > > > > Isn''t this just like: > > 6493923 nfsfind on ZFS filesystem quickly depletes memory in a 1GB system > > > > Which was introduced in b51(or 52) and fixed in snv_54. > > I''ve upgraded to snv54 now and even forced 500k dnlc entries in memory > (ncsize=500000,arc_reduce_dnlc_percent=0).. Let''s see how it copes with > that ;)Not too well.. Just went into swap hell.. and why is anon/exec&lib only using 5+2 MB when there''s about 50MB marked as Free?> arc_reduce_dnlc_percent/Darc_reduce_dnlc_percent: arc_reduce_dnlc_percent: 0> dnlc_nentries/Ddnlc_nentries: dnlc_nentries: 499126> freemem/Jfreemem: freemem: 9ea> ::memstatPage Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 248982 1945 97% Anon 730 5 0% Exec and libs 271 2 0% Page cache 38 0 0% Free (cachelist) 738 5 0% Free (freelist) 6108 47 2% Total 256867 2006 Physical 253028 1976> arc_reduce_dnlc_percent/W 0t3arc_reduce_dnlc_percent: 0 = 0x3> ::memstatPage Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 217511 1699 85% Anon 1502 11 1% Exec and libs 468 3 0% Page cache 48 0 0% Free (cachelist) 333 2 0% Free (freelist) 37005 289 14% Total 256867 2006 Physical 253028 1976> dnlc_nentries/Ddnlc_nentries: dnlc_nentries: 15183> freemem/Jfreemem: freemem: 9351> arc::print{ size = 0xc34aa00 p = 0x4000000 c = 0x4000000 c_min = 0x4000000 c_max = 0x5e114800> ARC_mru::print{ lsize = 0x4039400 size = 0x8bf9e00> ARC_mru_ghost::print{ lsize = 0x5a6c00 size = 0x5a6c00> ARC_mfu::print{ lsize = 0x2308800 size = 0x2fd4800> ARC_mfu_ghost::print{ lsize = 0xd8000 size = 0xd8000 Anything useful there? I did a savecore -L after this info was gathered (it was too slow to be useful before that.. had to wait a long time for mdb to show up), so a dump is available.. I also have a logfile of ~every minute with ::kmastat, ::memstat and dnlc_nentries from boot up until it started having problems, if someone knows what to look for.. Seems like Kernel used up around 84% steadily, then all of a sudden it started growing and within minutes Kernel was doing 97% and the machine more or less stopped moving.. It could be related to TSM starting backups (which it did about 1h before the "issue").. and after it had looked through whatever dnlc had cached, stuff went haywire.. </end theory> :) /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se
Jürgen Keil
2007-Jan-05 11:38 UTC
[zfs-discuss] Re: ZFS related (probably) hangs due to memory exhaustion(?) with snv53
> >Hmmm, so there is lots of evictable cache here (mostly in the MFU > >part of the cache)... could you make your core file available? > >I would like to take a look at it. > > Isn''t this just like: > 6493923 nfsfind on ZFS filesystem quickly depletes memory in a 1GB system > > Which was introduced in b51(or 52) and fixed in snv_54.Hmm, or like: 6483887 without direct management, arc ghost lists can run amok (which isn''t fixed at this time) See also this thread: http://www.opensolaris.org/jive/thread.jspa?messageID=67370 Mark had send me some test bits with a modified arc.c; it tried to evict ghost list entries when the arc cache is in no_grow state and the arc ghost lists consume too much memory. The main change was a new function arc_buf_hdr_alloc(), in arc.c that shrinks the ghost lists when the system is running out of memory: static arc_buf_hdr_t * arc_buf_hdr_alloc(spa_t *spa, int size) { arc_buf_hdr_t *hdr; if (arc.no_grow && arc.mru_ghost->size + arc.mfu_ghost->size > arc.c) { int64_t mru_over = arc.anon->size + arc.mru->size + arc.mru_ghost->size - arc.c; if (mru_over > 0 && arc.mru_ghost->size > 0) { int64_t todelete = MIN(arc.mru_ghost->lsize, mru_over); arc_evict_ghost(arc.mru_ghost, todelete); } else { int64_t todelete = MIN(arc.mfu_ghost->lsize, arc.mru_ghost->size + arc.mfu_ghost->size - arc.c); arc_evict_ghost(arc.mfu_ghost, todelete); } } ASSERT3U(size, >, 0); hdr = kmem_cache_alloc(hdr_cache, KM_SLEEP); ASSERT(BUF_EMPTY(hdr)); hdr->b_size = size; hdr->b_spa = spa; hdr->b_state = arc.anon; hdr->b_arc_access = 0; hdr->b_flags = 0; return (hdr); } This was then used by arc_buf_alloc(): arc_buf_t * arc_buf_alloc(spa_t *spa, int size, void *tag) { arc_buf_hdr_t *hdr; arc_buf_t *buf; hdr = arc_buf_hdr_alloc(spa, size); buf = kmem_cache_alloc(buf_cache, KM_SLEEP); buf->b_hdr = hdr; ... return (buf); } This message posted from opensolaris.org
Robert Milkowski
2007-Jan-05 11:49 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory exhaustion(?) with snv53
Hello Tomas, Friday, January 5, 2007, 4:00:53 AM, you wrote: T?> On 04 January, 2007 - Tomas ?gren sent me these 1,0K bytes:>> On 03 January, 2007 - Casper.Dik at Sun.COM sent me these 0,5K bytes: >> >> > >> > >Hmmm, so there is lots of evictable cache here (mostly in the MFU >> > >part of the cache)... could you make your core file available? >> > >I would like to take a look at it. >> > >> > Isn''t this just like: >> > 6493923 nfsfind on ZFS filesystem quickly depletes memory in a 1GB system >> > >> > Which was introduced in b51(or 52) and fixed in snv_54. >> >> I''ve upgraded to snv54 now and even forced 500k dnlc entries in memory >> (ncsize=500000,arc_reduce_dnlc_percent=0).. Let''s see how it copes with >> that ;)T?> Not too well.. Just went into swap hell.. and why is anon/exec&lib only T?> using 5+2 MB when there''s about 50MB marked as Free?>> arc_reduce_dnlc_percent/DT?> arc_reduce_dnlc_percent: T?> arc_reduce_dnlc_percent: 0>> dnlc_nentries/DT?> dnlc_nentries: T?> dnlc_nentries: 499126>> freemem/JT?> freemem: T?> freemem: 9ea>> ::memstatT?> Page Summary Pages MB %Tot T?> ------------ ---------------- ---------------- ---- T?> Kernel 248982 1945 97% T?> Anon 730 5 0% T?> Exec and libs 271 2 0% T?> Page cache 38 0 0% T?> Free (cachelist) 738 5 0% T?> Free (freelist) 6108 47 2% T?> Total 256867 2006 T?> Physical 253028 1976>> arc_reduce_dnlc_percent/W 0t3T?> arc_reduce_dnlc_percent: 0 = 0x3>> ::memstatT?> Page Summary Pages MB %Tot T?> ------------ ---------------- ---------------- ---- T?> Kernel 217511 1699 85% T?> Anon 1502 11 1% T?> Exec and libs 468 3 0% T?> Page cache 48 0 0% T?> Free (cachelist) 333 2 0% T?> Free (freelist) 37005 289 14% T?> Total 256867 2006 T?> Physical 253028 1976>> dnlc_nentries/DT?> dnlc_nentries: T?> dnlc_nentries: 15183>> freemem/JT?> freemem: T?> freemem: 9351>> arc::printT?> { T?> size = 0xc34aa00 T?> p = 0x4000000 T?> c = 0x4000000 T?> c_min = 0x4000000 T?> c_max = 0x5e114800>> ARC_mru::printT?> { T?> lsize = 0x4039400 T?> size = 0x8bf9e00>> ARC_mru_ghost::printT?> { T?> lsize = 0x5a6c00 T?> size = 0x5a6c00>> ARC_mfu::printT?> { T?> lsize = 0x2308800 T?> size = 0x2fd4800>> ARC_mfu_ghost::printT?> { T?> lsize = 0xd8000 T?> size = 0xd8000 T?> Anything useful there? I did a savecore -L after this info was gathered T?> (it was too slow to be useful before that.. had to wait a long time for T?> mdb to show up), so a dump is available.. T?> I also have a logfile of ~every minute with ::kmastat, ::memstat and T?> dnlc_nentries from boot up until it started having problems, if someone T?> knows what to look for.. T?> Seems like Kernel used up around 84% steadily, then all of a sudden it T?> started growing and within minutes Kernel was doing 97% and the machine T?> more or less stopped moving.. I saw the same behavior here when ncsize was increased from default. Try with default and lets see what will happen - if it works then it''s better than hung every an hour or so. -- Best regards, Robert mailto:rmilkowski at task.gda.pl http://milek.blogspot.com
Tomas Ögren
2007-Jan-05 12:20 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory exhaustion(?) with snv53
On 05 January, 2007 - Robert Milkowski sent me these 3,8K bytes:> Hello Tomas, > > I saw the same behavior here when ncsize was increased from default. > Try with default and lets see what will happen - if it works then it''s > better than hung every an hour or so.That''s still not the point.. It was fine with ncsize=500k (and all of it used) for a while.. then all of a sudden it just want haywire.. and when it freed up dnlc, I got back 250MB.. where''s the rest ~1750MB tied up? /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se
Mark Maybee
2007-Jan-05 16:24 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory exhaustion(?) with snv53
Thomas, This could be fragmentation in the meta-data caches. Could you print out the results of ::kmastat? -Mark Tomas ?gren wrote:> On 05 January, 2007 - Robert Milkowski sent me these 3,8K bytes: > >> Hello Tomas, >> >> I saw the same behavior here when ncsize was increased from default. >> Try with default and lets see what will happen - if it works then it''s >> better than hung every an hour or so. > > That''s still not the point.. It was fine with ncsize=500k (and all of it > used) for a while.. then all of a sudden it just want haywire.. and when > it freed up dnlc, I got back 250MB.. where''s the rest ~1750MB tied up? > > /Tomas
Tomas Ögren
2007-Jan-05 17:47 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory exhaustion(?) with snv53
On 05 January, 2007 - Mark Maybee sent me these 0,8K bytes:> Thomas, > > This could be fragmentation in the meta-data caches. Could you > print out the results of ::kmastat?http://www.acc.umu.se/~stric/tmp/zfs-dumps.tar.bz2 memstat, kmastat and dnlc_nentries from 10 minutes after boot up until the "near death experience".. I''ve got vmcore as well if needed.. /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se
Mark Maybee
2007-Jan-05 19:19 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory exhaustion(?) with snv53
So it looks like this data does not include ::kmastat info from *after* you reset arc_reduce_dnlc_percent. Can I get that? What I suspect is happening: 1 with your large ncsize, you eventually ran the machine out of memory because (currently) the arc is not accounting for the space consumed by "auxiliary" caches (dnode_t, etc.). 2 the arc could not reduce at this point since almost all of its memory was tied up by the dnlc refs. 3 when you eventually allowed the arc to reduce the dnlc size, it managed to free up some space, but much of this did not "appear" because it was tied up in slabs in the auxiliary caches (fragmentation). We are working on a fix for number 1 above. You should *not* be setting arc_reduce_dnlc_percent to zero, even if you want a large number of dnlc entries. You are tying the arc hands here, so it has no ability to reduce its size. Number 3 is the most difficult issue. We are looking into that at the moment as well. -Mark Tomas ?gren wrote:> On 05 January, 2007 - Mark Maybee sent me these 0,8K bytes: > >> Thomas, >> >> This could be fragmentation in the meta-data caches. Could you >> print out the results of ::kmastat? > > http://www.acc.umu.se/~stric/tmp/zfs-dumps.tar.bz2 > > memstat, kmastat and dnlc_nentries from 10 minutes after boot up until > the "near death experience".. > > I''ve got vmcore as well if needed.. > > /Tomas
Tomas Ögren
2007-Jan-05 19:30 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory exhaustion(?) with snv53
On 05 January, 2007 - Mark Maybee sent me these 1,5K bytes:> So it looks like this data does not include ::kmastat info from *after* > you reset arc_reduce_dnlc_percent. Can I get that?Yeah, attached. (although about 18 hours after the others)> What I suspect is happening: > 1 with your large ncsize, you eventually ran the machine out > of memory because (currently) the arc is not accounting for > the space consumed by "auxiliary" caches (dnode_t, etc.). > 2 the arc could not reduce at this point since almost all of > its memory was tied up by the dnlc refs. > 3 when you eventually allowed the arc to reduce the dnlc size, > it managed to free up some space, but much of this did not > "appear" because it was tied up in slabs in the auxiliary > caches (fragmentation). > > We are working on a fix for number 1 above.Great!> You should *not* be setting arc_reduce_dnlc_percent to zero, even if > you want a large number of dnlc entries. You are tying the arc hands > here, so it has no ability to reduce its size. > Number 3 is the most difficult issue. We are looking into that at the > moment as well.Any idea where all the memory is going? I sure hope that 500k dnlc entries (+dnode_t''s etc belonging to that) isn''t using up about 2GB ram..? /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se -------------- next part -------------- cache buf buf buf memory alloc alloc name size in use total in use succeed fail ------------------------- ------ ------ ------ --------- --------- ----- kmem_magazine_1 16 66933 68072 1097728 1661005 0 kmem_magazine_3 32 5156 185166 5971968 3651757 0 kmem_magazine_7 64 128279 167132 10780672 833454 0 kmem_magazine_15 128 95475 95508 12419072 772265 0 kmem_magazine_31 256 10766 10850 2867200 446593 0 kmem_magazine_47 384 394 1344 524288 268565 0 kmem_magazine_63 512 15 1395 761856 892021 0 kmem_magazine_95 768 136 220 180224 1724 0 kmem_magazine_143 1152 561 567 663552 561 0 kmem_slab_cache 56 31587 57130 3227648 8832570 0 kmem_bufctl_cache 24 259255 275268 6651904 11041532 0 kmem_bufctl_audit_cache 128 0 0 0 0 0 kmem_va_8192 8192 156227 182080 1491599360 563808 0 kmem_va_16384 16384 6760 24016 393478144 2608303 0 kmem_va_24576 24576 30 90 2359296 26820 0 kmem_va_32768 32768 4102 9864 323223552 8220474 0 kmem_va_40960 40960 38 264 11534336 237265 0 kmem_va_49152 49152 5 20 1048576 3009 0 kmem_va_57344 57344 98 428 28049408 543538 0 kmem_va_65536 65536 477 2992 196083712 835607 0 kmem_alloc_8 8 8697 13221 106496 7731684 0 kmem_alloc_16 16 9269 12700 204800 562429206 0 kmem_alloc_24 24 8622 13221 319488 148101813 0 kmem_alloc_32 32 32929 163322 5267456 673202709 0 kmem_alloc_40 40 44236 55622 2244608 57956587 0 kmem_alloc_48 48 10631 62361 3022848 66508294 0 kmem_alloc_56 56 16033 96570 5455872 3931601 0 kmem_alloc_64 64 18570 157607 10166272 46889692 0 kmem_alloc_80 80 135312 622564 50495488 40453962 0 kmem_alloc_96 96 179070 291396 28418048 23482483 0 kmem_alloc_112 112 50004 135072 15368192 98592237 0 kmem_alloc_128 128 11253 23058 2998272 8022665 0 kmem_alloc_160 160 3172 7800 1277952 1022154 0 kmem_alloc_192 192 298 924 180224 2142183 0 kmem_alloc_224 224 279 360 81920 1391779 0 kmem_alloc_256 256 7535 24800 6553600 188838186 0 kmem_alloc_320 320 1689 1750 573440 560146 0 kmem_alloc_384 384 274 1827 712704 60590097 0 kmem_alloc_448 448 2706 2772 1261568 312999151 0 kmem_alloc_512 512 316 690 376832 16436341 0 kmem_alloc_640 640 88 12732 8691712 517476456 0 kmem_alloc_768 768 23 40 32768 180648 0 kmem_alloc_896 896 27 36 32768 307859 0 kmem_alloc_1152 1152 721 798 933888 3858662 0 kmem_alloc_1344 1344 55 66 90112 451770 0 kmem_alloc_1600 1600 30 40 65536 3009471 0 kmem_alloc_2048 2048 98 120 245760 2764046 0 kmem_alloc_2688 2688 103 114 311296 762011 0 kmem_alloc_4096 4096 53 58 237568 173305 0 kmem_alloc_8192 8192 5157 5353 43851776 1335040 0 kmem_alloc_12288 12288 15 32 393216 14583 0 kmem_alloc_16384 16384 38 39 638976 2034 0 streams_mblk 64 18468 18542 1196032 530378743 0 streams_dblk_16 128 55 126 16384 779280 0 streams_dblk_80 192 292 462 90112 4522693 0 streams_dblk_144 256 2 62 16384 489060682 0 streams_dblk_208 320 265 375 122880 251539 0 streams_dblk_272 384 7 84 32768 97143 0 streams_dblk_336 448 0 18 8192 6065 0 streams_dblk_528 640 0 24 16384 21748 0 streams_dblk_1040 1152 1 35 40960 18605 0 streams_dblk_1488 1600 0 10 16384 5225 0 streams_dblk_1936 2048 1282 17092 35004416 352153376 0 streams_dblk_2576 2688 0 12 32768 2211458 0 streams_dblk_3920 4032 0 4 16384 8864 0 streams_dblk_8192 112 0 63 8192 94593 0 streams_dblk_12112 12224 0 4 49152 46740 0 streams_dblk_16384 112 0 63 8192 96031 0 streams_dblk_20304 20416 0 4 81920 28034 0 streams_dblk_24576 112 0 63 8192 50449 0 streams_dblk_28496 28608 0 4 114688 10747 0 streams_dblk_32768 112 0 126 16384 19959114 0 streams_dblk_36688 36800 0 38 1400832 928268 0 streams_dblk_40960 112 0 63 8192 1110 0 streams_dblk_44880 44992 0 4 180224 808 0 streams_dblk_49152 112 0 63 8192 1550 0 streams_dblk_53072 53184 0 4 212992 1301 0 streams_dblk_57344 112 0 63 8192 620 0 streams_dblk_61264 61376 0 4 245760 513 0 streams_dblk_65536 112 0 63 8192 19819 0 streams_dblk_69456 69568 0 4 278528 1171 0 streams_dblk_73728 112 0 63 8192 304 0 streams_dblk_esb 112 0 126 16384 20804 0 streams_fthdr 264 0 0 0 0 0 streams_ftblk 232 0 0 0 0 0 multidata 248 0 0 0 0 0 multidata_pdslab 7112 0 0 0 0 0 multidata_pattbl 32 0 0 0 0 0 log_cons_cache 48 8 169 8192 41 0 taskq_ent_cache 56 2475 3480 196608 27572807 0 taskq_cache 216 70 111 24576 92 0 id32_cache 32 0 0 0 0 0 bp_map_16384 16384 0 32 524288 762 0 bp_map_32768 32768 0 16 524288 2563 0 bp_map_49152 49152 0 10 524288 2211 0 bp_map_65536 65536 0 8 524288 1874 0 bp_map_81920 81920 0 6 524288 1705 0 bp_map_98304 98304 0 5 524288 1409 0 bp_map_114688 114688 0 4 524288 1092 0 bp_map_131072 131072 0 4 524288 1444 0 memseg_cache 112 0 0 0 0 0 mod_hash_entries 24 249 678 16384 4185 0 ipp_mod 304 0 0 0 0 0 ipp_action 368 0 0 0 0 0 ipp_packet 64 0 0 0 0 0 mmuctxdom_cache 192 2 42 8192 2 0 sfmmuid_cache 184 49 84 16384 22032 0 sfmmu_tsbinfo_cache 64 48 127 8192 49826 0 sfmmu_tsb8k_cache 8192 0 0 0 0 0 sfmmu_tsb_cache 8192 28 35 286720 24083 0 sfmmu8_cache 312 39105 87360 27525120 8086400 0 sfmmu1_cache 88 3557 3588 319488 193984 0 pa_hment_cache 64 43 127 8192 27925 0 ism_blk_cache 272 0 0 0 0 0 ism_ment_cache 32 0 0 0 0 0 seg_cache 72 2892 3390 245760 2149118 0 dev_info_node_cache 488 174 320 163840 1217 0 segkp_8192 8192 42 48 393216 11700 0 segkp_16384 16384 0 0 0 0 0 segkp_24576 24576 0 0 0 0 0 segkp_32768 32768 547 600 19660800 4234 0 segkp_40960 40960 0 0 0 0 0 umem_np_8192 8192 0 32 262144 222 0 umem_np_16384 16384 0 16 262144 100 0 umem_np_24576 24576 0 0 0 0 0 umem_np_32768 32768 0 8 262144 9 0 umem_np_40960 40960 0 0 0 0 0 umem_np_49152 49152 0 0 0 0 0 umem_np_57344 57344 0 0 0 0 0 umem_np_65536 65536 0 4 262144 9 0 thread_cache 800 231 260 212992 29167 0 wbuf32_cache 512 228 270 147456 29187 0 wbuf64_cache 1024 3 14 16384 738 0 lwp_cache 912 231 264 270336 1136 0 turnstile_cache 64 541 635 40960 34817 0 tslabel_cache 48 2 169 8192 2 0 cred_cache 172 91 138 24576 2574 0 rctl_cache 40 741 1015 40960 145396 0 rctl_val_cache 64 1454 1905 122880 317132 0 task_cache 104 34 78 8192 410 0 cyclic_id_cache 64 10 127 8192 31 0 dnlc_space_cache 24 19 339 8192 6062 0 vn_cache 240 405388 657696 173801472 948191 0 vsk_anchor_cache 40 15 203 8192 44 0 file_cache 56 402 580 32768 827834 0 stream_head_cache 368 195 220 81920 74333 0 queue_cache 656 508 540 368640 212667 0 syncq_cache 160 11 50 8192 29 0 qband_cache 64 2 127 8192 2 0 linkinfo_cache 48 4 169 8192 4 0 ciputctrl_cache 128 4 63 8192 4 0 serializer_cache 64 26 127 8192 151 0 as_cache 224 48 72 16384 22031 0 marker_cache 128 0 63 8192 156031 0 anon_cache 48 15062 16224 786432 3343368 0 anonmap_cache 64 1754 2159 139264 437468 0 segvn_cache 104 2892 3432 360448 1072573 0 flk_edges 48 0 169 8192 1 0 fdb_cache 104 0 0 0 0 0 timer_cache 136 1 59 8192 2 0 physio_buf_cache 248 0 32 8192 305 0 snode_cache 152 363 424 65536 290321 0 ufs_inode_cache 368 58473 75042 27942912 118311 0 directio_buf_cache 272 0 0 0 0 0 lufs_save 24 0 339 8192 45759 0 lufs_bufs 256 0 31 8192 47055 0 lufs_mapentry_cache 112 44 1656 188416 1067289 0 pcisch0_dvma_8192 8192 44 1008 8257536 269346609 0 pcisch1_dvma_8192 8192 1282 1328 10878976 312538496 0 fctl_cache 112 0 72 8192 9 0 fp1_cache 728 1 11 8192 6 0 fcp1_cache 1224 2 39 49152 1440466 0 dv_node_cache 128 93 189 24576 839 0 sdev_node_cache 200 582 600 122880 762 0 clnt_clts_endpnt_cache 88 2 92 8192 190 0 md_stripe_parent 96 0 252 24576 1269327 0 md_stripe_child 312 0 260 81920 1398467 0 md_mirror_parent 160 0 250 40960 803301 0 md_mirror_child 304 0 260 81920 1397128 0 md_mirror_wow 16440 0 8 139264 18143 0 kcf_sreq_cache 48 0 0 0 0 0 kcf_areq_cache 272 0 0 0 0 0 kcf_context_cache 88 0 0 0 0 0 ipsec_actions 72 0 113 8192 1920 0 ipsec_selectors 80 0 0 0 0 0 ipsec_policy 72 0 0 0 0 0 ipsec_info 336 0 24 8192 2587 0 ip_minor_arena_1 1 83 128 128 53719 0 ipcl_conn_cache 488 39 60 32768 12549 0 ipcl_tcpconn_cache 1696 47 72 131072 83856 0 radix_mask 32 1 254 8192 1 0 radix_node 112 2 72 8192 2 0 rt_entry 160 3 50 8192 4 0 ire_cache 344 56 92 32768 2534 0 tcp_timercache 88 117 184 16384 13129 0 tcp_sack_info_cache 80 19 101 8192 40546 0 tcp_iphc_cache 120 46 134 16384 82242 0 squeue_cache 136 2 42 8192 2 0 sctp_conn_cache 2264 1 7 16384 1 0 sctp_faddr_cache 168 0 0 0 0 0 sctp_set_cache 24 0 0 0 0 0 sctp_ftsn_set_cache 16 0 0 0 0 0 ire_gw_secattr_cache 32 0 0 0 0 0 sctpsock 616 0 0 0 0 0 sctp_assoc 64 0 0 0 0 0 socktpi_cache 456 20 34 16384 3523 0 socktpi_unix_cache 456 14 34 16384 124 0 mac_impl_cache 384 0 0 0 0 0 dls_cache 152 0 0 0 0 0 soft_ring_cache 176 0 0 0 0 0 dls_vlan_cache 48 0 0 0 0 0 dls_link_cache 368 0 0 0 0 0 dld_ctl_1 1 0 0 0 0 0 drv_secobj_cache 296 0 0 0 0 0 dld_str_cache 272 0 29 8192 1 0 udp_cache 416 34 54 24576 12535 0 process_cache 3080 50 60 196608 11452 0 exacct_object_cache 40 0 0 0 0 0 ch_private_cache 13200 2 2 32768 2 0 hal0_cache 464 0 17 8192 1 0 kssl_cache 1560 0 0 0 0 0 tl_cache 432 39 54 24576 393 0 keysock_1 1 0 0 0 0 0 spdsock_1 1 0 64 64 1 0 fnode_cache 176 5 42 8192 34 0 pipe_cache 320 24 50 16384 9219 0 namefs_inodes_1 1 19 64 64 49 0 port_cache 80 3 101 8192 3 0 ip_minor_1 1 0 0 0 0 0 ar_minor_1 1 0 0 0 0 0 lnode_cache 32 2 254 8192 2 0 zio_buf_512 512 137801 294975 161095680 43660052 0 zio_buf_1024 1024 7 80 81920 7730712 0 zio_buf_1536 1536 4 20 32768 4319811 0 zio_buf_2048 2048 6 16 32768 840840 0 zio_buf_2560 2560 1 12 32768 437525 0 zio_buf_3072 3072 7 16 49152 658297 0 zio_buf_3584 3584 2 9 32768 647863 0 zio_buf_4096 4096 2 6 24576 523423 0 zio_buf_5120 5120 3 16 81920 3124059 0 zio_buf_6144 6144 2 8 49152 857598 0 zio_buf_7168 7168 1 336 2408448 104619346 0 zio_buf_8192 8192 2 5 40960 133985 0 zio_buf_10240 10240 2 8 81920 99216 0 zio_buf_12288 12288 0 6 73728 41462 0 zio_buf_14336 14336 0 116 1662976 13512747 0 zio_buf_16384 16384 6692 6697 109723648 5877279 0 zio_buf_20480 20480 1 20 409600 8538438 0 zio_buf_24576 24576 1 5 122880 12157 0 zio_buf_28672 28672 1 42 1204224 7455093 0 zio_buf_32768 32768 3701 4101 134381568 35395387 0 zio_buf_40960 40960 1 22 901120 1086580 0 zio_buf_49152 49152 3 5 245760 20708 0 zio_buf_57344 57344 2 4 229376 20091 0 zio_buf_65536 65536 365 477 31260672 2831209 0 zio_buf_73728 73728 0 3 221184 12129 0 zio_buf_81920 81920 0 3 245760 7014 0 zio_buf_90112 90112 0 4 360448 5552 0 zio_buf_98304 98304 0 3 294912 5997 0 zio_buf_106496 106496 0 3 319488 9767 0 zio_buf_114688 114688 0 2 229376 4570 0 zio_buf_122880 122880 0 3 368640 3911 0 zio_buf_131072 131072 0 2 262144 136994 0 dmu_buf_impl_t 328 145260 622392 212443136 65461261 0 dnode_t 640 137799 512508 349872128 37995548 0 arc_buf_hdr_t 144 21457 43344 6340608 35018385 0 arc_buf_t 40 10823 19285 778240 60847028 0 zil_lwb_cache 200 0 0 0 0 0 zfs_znode_cache 200 137763 568040 116334592 35683478 0 mpt1_cache 536 7 315 172032 93673786 0 mpt0_cache 536 7 315 172032 92907412 0 md_raid_parent 120 0 0 0 0 0 md_raid_child 1040 0 0 0 0 0 md_raid_cbufs 376 0 0 0 0 0 md_trans_parent 80 0 0 0 0 0 md_trans_child 248 0 0 0 0 0 glm0_cache 472 4 17 8192 1294 0 icmp_minor_1 1 0 0 0 0 0 authkern_cache 72 0 113 8192 32203 0 authloopback_cache 72 0 0 0 0 0 authdes_cache_handle 80 0 0 0 0 0 rnode_cache 648 4 144 98304 16551 0 nfs_access_cache 56 1 1885 106496 2440 0 client_handle_cache 32 4 254 8192 43 0 rnode4_cache 960 0 0 0 0 0 svnode_cache 40 0 0 0 0 0 nfs4_access_cache 56 0 0 0 0 0 client_handle4_cache 32 0 0 0 0 0 nfs4_ace4vals_cache 48 0 0 0 0 0 nfs4_ace4_list_cache 264 0 0 0 0 0 NFS_idmap_cache 48 0 169 8192 6 0 nfslog_small_rec 512 0 0 0 0 0 nfslog_medium_rec 8192 0 0 0 0 0 nfslog_large_rec 32768 0 0 0 0 0 exi_cache_handle 40 19 203 8192 59 0 dtrace_state_cache 2048 0 4 8192 9 0 lm_vnode 184 0 0 0 0 0 lm_xprt 32 0 0 0 0 0 lm_sysid 160 0 0 0 0 0 lm_client 128 0 0 0 0 0 lm_async 32 0 0 0 0 0 lm_sleep 96 0 0 0 0 0 lm_config 80 2 101 8192 2 0 pty_map 64 3 127 8192 7 0 glm1_cache 472 0 17 8192 152 0 Client_entry_cache 744 0 10 8192 1 0 OpenOwner_entry_cache 528 0 0 0 0 0 OpenStateID_entry_cache 256 0 0 0 0 0 LockStateID_entry_cache 504 0 0 0 0 0 Lockowner_entry_cache 176 0 0 0 0 0 File_entry_cache 288 0 0 0 0 0 DelegStateID_entry_cache 248 0 0 0 0 0 md_softpart_parent 88 0 0 0 0 0 md_softpart_child 304 0 0 0 0 0 fcip1_cache 728 0 0 0 0 0 fcip1_sendup_cache 24 0 0 0 0 0 fcsm_job_cache 104 0 0 0 0 0 fcsm1_cmd_cache 712 0 0 0 0 0 crypto_session_cache 96 0 0 0 0 0 aggr_port_cache 1056 0 0 0 0 0 aggr_grp_cache 680 0 0 0 0 0 ------------------------- ------ ------ ------ --------- --------- ----- Total [static] 434176 109051 0 Total [hat_memload] 27525120 8086400 0 Total [kmem_msb] 45146112 28402047 0 Total [kmem_va] 2447376384 13038824 0 Total [kmem_default] 1569161216 667252352 0 Total [bp_map] 4194304 13060 0 Total [kmem_tsb_default] 286720 24083 0 Total [hat_memload1] 319488 193984 0 Total [umem_np] 1048576 340 0 Total [segkp] 20054016 15934 0 Total [pcisch0_dvma] 8257536 269346609 0 Total [pcisch1_dvma] 10878976 312538496 0 Total [ip_minor_arena] 128 53719 0 Total [spdsock] 64 1 0 Total [namefs_inodes] 64 49 0 ------------------------- ------ ------ ------ --------- --------- ----- vmem memory memory memory alloc alloc name in use total import succeed fail ------------------------- --------- ---------- --------- --------- ----- heap 2632138752 4398046511104 0 57256 0 vmem_metadata 33038336 33292288 33292288 3882 0 vmem_seg 31260672 31260672 31260672 3816 0 vmem_hash 1450752 1466368 1466368 74 0 vmem_vmem 295800 346096 311296 101 0 static 458752 458752 458752 55 0 static_alloc 8320 16384 16384 3 0 hat_memload 27525120 27525120 27525120 3434 0 kstat 266568 278528 212992 944 0 kmem_metadata 48144384 48234496 48234496 6878 0 kmem_msb 45146112 45146112 45146112 6559 0 kmem_cache 202872 221184 221184 380 0 kmem_hash 2770944 2777088 2777088 1638 0 kmem_log 131488 139264 139264 6 0 kmem_firewall_va 37429248 37429248 37429248 23998 0 kmem_firewall 0 0 0 0 0 kmem_oversize 36777327 37429248 37429248 24004 0 mod_sysfile 892 8192 8192 25 0 kmem_va 2452135936 2452135936 2452135936 27933 0 kmem_default 1569243136 1569243136 1569243136 8579884 0 little_endian 179168 204800 204800 487 0 big_endian 4455340 4931584 4931584 6631 0 bp_map 4194304 4194304 4194304 713 0 ksyms 1717794 1744896 1744896 305 0 ctf 205377 212992 212992 307 0 kmem_tsb 4194304 4194304 4194304 1 0 kmem_tsb_default 745472 4194304 4194304 5965 0 hat_memload1 319488 319488 319488 111 0 umem_np 1179648 1179648 1179648 12 0 heap32 2086656 134217728 0 506 0 id32 0 0 0 0 0 module_data 1889384 2293760 2031616 399 0 promplat 0 0 0 464 0 trapstat 0 0 0 0 0 heaptext 39337984 134217728 0 122 0 module_text 7705380 7831552 5775360 303 0 logminor_space 24 262137 0 57 0 taskq_id_arena 34 2147483647 0 56 0 segkp 20054016 2147483648 0 2466 0 rctl_ids 32 32767 0 32 0 zoneid_space 0 9998 0 0 0 taskid_space 34 999999 0 410 0 pool_ids 0 999998 0 0 0 contracts 36 2147483646 0 446 0 regspec 884736 5368709120 0 144 0 pcisch0_dvma 8257536 1040187392 0 121529 0 pcisch1_dvma 15728640 1040187392 0 73106 0 ip_minor_arena 128 262140 0 2 0 dld_ctl 0 4294967295 0 0 0 dld_minor_arena 1 4294967295 0 1 0 tl_minor_space 39 262138 0 367 0 keysock 0 4294967295 0 0 0 spdsock 64 4294967295 0 1 0 namefs_inodes 64 65536 0 1 0 ip_minor 0 262142 0 0 0 ar_minor 0 262142 0 0 0 devfsadm_event_channel 1 101 0 1 0 devfsadm_event_channel 1 2 0 1 0 syseventconfd_event_channel 0 101 0 0 0 syseventconfd_event_channel 1 2 0 1 0 syseventd_channel 4 101 0 4 0 syseventd_channel 1 2 0 1 0 icmp_minor 0 262142 0 0 0 lmsysid_space 1 16383 0 1 0 dtrace 948 4294967295 0 61355 0 dtrace_minor 0 4294967293 0 9 0 ptms_minor 3 16 0 7 0 heaptext_holesrc_15 368640 2097152 0 27 0 heaptext_hole_15 320364 368640 368640 72 0 heaptext_holesrc_14 540672 2097152 0 14 0 heaptext_hole_14 501880 540672 540672 33 0 Client_id_space 0 131072 0 1 0 OpenOwner_id_space 0 1048576 0 0 0 OpenStateID_id_space 0 1048576 0 0 0 LockStateID_id_space 0 1048576 0 0 0 Lockowner_id_space 0 1048576 0 0 0 DelegStateID_id_space 0 1048576 0 0 0 logdmux_minor 0 256 0 0 0 heaptext_holesrc_13 0 2097152 0 1 0 heaptext_hole_13 0 0 0 1 0 module_text_holesrc 1409024 4194304 0 28 0 heaptext_hole_0 1374100 1409024 1409024 68 0 heaptext_holesrc_16 557056 2097152 0 16 0 heaptext_hole_16 527752 557056 557056 48 0 crypto 0 16 0 3 0 ------------------------- --------- ---------- --------- --------- -----
Mark Maybee
2007-Jan-05 20:25 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory exhaustion(?) with snv53
Tomas ?gren wrote:> On 05 January, 2007 - Mark Maybee sent me these 1,5K bytes: > >> So it looks like this data does not include ::kmastat info from *after* >> you reset arc_reduce_dnlc_percent. Can I get that? > > Yeah, attached. (although about 18 hours after the others) >Excellent, this confirms #3 below.>> What I suspect is happening: >> 1 with your large ncsize, you eventually ran the machine out >> of memory because (currently) the arc is not accounting for >> the space consumed by "auxiliary" caches (dnode_t, etc.). >> 2 the arc could not reduce at this point since almost all of >> its memory was tied up by the dnlc refs. >> 3 when you eventually allowed the arc to reduce the dnlc size, >> it managed to free up some space, but much of this did not >> "appear" because it was tied up in slabs in the auxiliary >> caches (fragmentation). >> >> We are working on a fix for number 1 above. > > Great! > >> You should *not* be setting arc_reduce_dnlc_percent to zero, even if >> you want a large number of dnlc entries. You are tying the arc hands >> here, so it has no ability to reduce its size. >> Number 3 is the most difficult issue. We are looking into that at the >> moment as well. > > Any idea where all the memory is going? I sure hope that 500k dnlc > entries (+dnode_t''s etc belonging to that) isn''t using up about 2GB > ram..? >Actually, thats pretty much what is happening: 500k dnlc => 170MB in the vnodes (vn_cache) + 320MB in znode_phys data (zio_buf_512) + 382MB in dnode_phys data (zio_buf_16384) + 208MB in dmu bufs (dmu_buf_impl_t) + 400MB in dnodes (dnode_t) + 120MB in znodes (zfs_znode_cache) --------- total 1600MB These numbers come from the last ::kmastat you ran before reducing the DNLC size. Note below that much of this space is still consumed by these caches, even after the DNLC has dropped it references. This is largely due to fragmentation in the caches.> /Tomas > > > ------------------------------------------------------------------------ > > cache buf buf buf memory alloc alloc > name size in use total in use succeed fail > ------------------------- ------ ------ ------ --------- --------- ----- > vn_cache 240 405388 657696 173801472 948191 0...> zio_buf_512 512 137801 294975 161095680 43660052 0...> zio_buf_16384 16384 6692 6697 109723648 5877279 0...> dmu_buf_impl_t 328 145260 622392 212443136 65461261 0 > dnode_t 640 137799 512508 349872128 37995548 0...> zfs_znode_cache 200 137763 568040 116334592 35683478 0
Tomas Ögren
2007-Jan-05 20:30 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory exhaustion(?) with snv53
On 05 January, 2007 - Tomas ?gren sent me these 33K bytes:> On 05 January, 2007 - Mark Maybee sent me these 1,5K bytes: > > > So it looks like this data does not include ::kmastat info from *after* > > you reset arc_reduce_dnlc_percent. Can I get that? > > Yeah, attached. (although about 18 hours after the others)Remembered that I have a vmcore of just after I changed.. but mdb has been eating 40 cpu minutes now and has just produced 40 lines out of 417 or so.. (and complained about some slabs/caches being out of sync with itself), so no dice. /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se
Tomas Ögren
2007-Jan-05 21:07 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory exhaustion(?) with snv53
On 05 January, 2007 - Mark Maybee sent me these 2,9K bytes:> Tomas ?gren wrote: > >On 05 January, 2007 - Mark Maybee sent me these 1,5K bytes: > > > >>So it looks like this data does not include ::kmastat info from *after* > >>you reset arc_reduce_dnlc_percent. Can I get that? > > > >Yeah, attached. (although about 18 hours after the others) > > > Excellent, this confirms #3 below.Theories that match reality is a big plus :)> >>What I suspect is happening: > >> 1 with your large ncsize, you eventually ran the machine out > >> of memory because (currently) the arc is not accounting for > >> the space consumed by "auxiliary" caches (dnode_t, etc.). > >> 2 the arc could not reduce at this point since almost all of > >> its memory was tied up by the dnlc refs. > >> 3 when you eventually allowed the arc to reduce the dnlc size, > >> it managed to free up some space, but much of this did not > >> "appear" because it was tied up in slabs in the auxiliary > >> caches (fragmentation).> >Any idea where all the memory is going? I sure hope that 500k dnlc > >entries (+dnode_t''s etc belonging to that) isn''t using up about 2GB > >ram..? > > > Actually, thats pretty much what is happening: > 500k dnlc => 170MB in the vnodes (vn_cache) > + 320MB in znode_phys data (zio_buf_512) > + 382MB in dnode_phys data (zio_buf_16384) > + 208MB in dmu bufs (dmu_buf_impl_t) > + 400MB in dnodes (dnode_t) > + 120MB in znodes (zfs_znode_cache) > --------- > total 1600MB > > These numbers come from the last ::kmastat you ran before reducing the > DNLC size. Note below that much of this space is still consumed by > these caches, even after the DNLC has dropped it references. This is > largely due to fragmentation in the caches.http://www.acc.umu.se/~stric/tmp/dnlc-plot.png They seem kinda related.. (buffers are the ones you mentioned here).. but the pike at the end doesn''t look very nice.. Also, about 3kB per entry is slightly over those 64 bytes that UFS uses according to the docs, which isn''t too promising for a fileserver with less than 32GB ram right now :)> >------------------------------------------------------------------------ > > > >cache buf buf buf memory alloc alloc > >name size in use total in use succeed fail > >------------------------- ------ ------ ------ --------- --------- ----- > >vn_cache 240 405388 657696 173801472 948191 0 > ... > >zio_buf_512 512 137801 294975 161095680 43660052 0 > ... > >zio_buf_16384 16384 6692 6697 109723648 5877279 0 > ... > >dmu_buf_impl_t 328 145260 622392 212443136 65461261 0 > >dnode_t 640 137799 512508 349872128 37995548 0 > ... > >zfs_znode_cache 200 137763 568040 116334592 35683478 0I can continue to do plots/dumps of these metrics.. I''ll try locking in 200k entries or so and see what happens.. /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se
Tomas Ögren
2007-Jan-07 13:41 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory exhaustion(?) with snv53
On 05 January, 2007 - Tomas ?gren sent me these 3,3K bytes:> > These numbers come from the last ::kmastat you ran before reducing the > > DNLC size. Note below that much of this space is still consumed by > > these caches, even after the DNLC has dropped it references. This is > > largely due to fragmentation in the caches. > > I can continue to do plots/dumps of these metrics.. I''ll try locking in > 200k entries or so and see what happens..http://www.acc.umu.se/~stric/tmp/dnlc-plot2.png This is after 1.5 day.. X axis is "kinda" minutes with a scaling error of 1.5-2.. Both spikes are from around when backups run, which isn''t too surprising.. Does seem to be some fragmentation, yes.. /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se
Tomas Ögren
2007-Jan-07 20:28 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory exhaustion(?) with snv53
On 07 January, 2007 - Tomas ?gren sent me these 1,0K bytes:> On 05 January, 2007 - Tomas ?gren sent me these 3,3K bytes: > > > > These numbers come from the last ::kmastat you ran before reducing the > > > DNLC size. Note below that much of this space is still consumed by > > > these caches, even after the DNLC has dropped it references. This is > > > largely due to fragmentation in the caches. > > > > I can continue to do plots/dumps of these metrics.. I''ll try locking in > > 200k entries or so and see what happens.. > > http://www.acc.umu.se/~stric/tmp/dnlc-plot2.pngUpdated.> This is after 1.5 day.. X axis is "kinda" minutes with a scaling error > of 1.5-2.. Both spikes are from around when backups run, which isn''t too > surprising.. Does seem to be some fragmentation, yes..And now it got another spike which killed it.. Backing off on the "forcing dnlc" practice right now.. Seems like some fragmentation issues has to be solved before that''s a viable option.. /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se
Tomas Ögren
2007-Jan-10 16:26 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory exhaustion(?)
Hello. Having some hangs on a snv53 machine which is quite probably ZFS+NFS related, since that''s all the machine do ;) The machine is a 2x750MHz Blade1000 with 2GB ram, using a SysKonnect 9821 GigE card (with their 8.19.1.3 skge driver) and two HP branded MPT SCSI cards. Normal load is pretty much "read all you can" with misc tarballs and isos since it''s an NFS backend to our caching http/ftp cluster delivering free software to the world. What happens is that the machine just stops responding.. it can respond to ping for a while (while userland is non-responsive, including console) but after a while, that stops too.. Produced a panic to get a dump and tried ::memstat; unterweser:/scratch/070103# mdb unix.0 vmcore.0 Loading modules: [ unix krtld genunix specfs dtrace ufs scsi_vhci pcisch ssd fcp fctl qlc md ip hook neti sctp arp usba s1394 nca lofs zfs random sd nfs ptm cpc ]> ::memstatPage Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 250919 1960 98% Anon 888 6 0% Exec and libs 247 1 0% Page cache 38 0 0% Free (cachelist) 405 3 0% Free (freelist) 4370 34 2% Total 256867 2006 Physical 253028 1976 That doesn''t seem too healthy to me.. probably something kernely eating up everything and the machine is just swapping to death or something.. A dump from live kernel with mdb -k after 1.5h uptime; Page Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 212310 1658 83% Anon 11307 88 4% Exec and libs 2418 18 1% Page cache 18400 143 7% Free (cachelist) 4383 34 2% Free (freelist) 8049 62 3% The tweaks I have are: set ncsize = 500000 set nfs:nrnode = 50 set zfs:zil_disable=1 set zfs:zfs_vdev_cache_bshift=14 set zfs:zfs_vdev_cache_size=0 Which according to ::kmem_cache results in about: 0000030002e30008 dmu_buf_impl_t 0000 000000 328 487728 0000030002e30288 dnode_t 0000 000000 640 453204 0000030002e30508 arc_buf_hdr_t 0000 000000 144 103544 0000030002e30788 arc_buf_t 0000 000000 40 36743 0000030002e30a08 zil_lwb_cache 0000 000000 200 0 0000030002e30c88 zfs_znode_cache 0000 000000 200 453200 but those buffers equal to about 550MB.. dnlc_nentries on the hung has gone down to 15000.. (where are the rest of the ~450k-15k dnode/znodes hanging out?) Hung kernel:> arc::print{ anon = ARC_anon mru = ARC_mru mru_ghost = ARC_mru_ghost mfu = ARC_mfu mfu_ghost = ARC_mfu_ghost size = 0x358a0600 p = 0x4000000 c = 0x4000000 c_min = 0x4000000 c_max = 0x5e114800 hits = 0xbc860fd misses = 0x2f296e1 deleted = 0x1d88739 recycle_miss = 0xf7f30c mutex_miss = 0x24b13d evict_skip = 0x21501d02 hash_elements = 0x27f97 hash_elements_max = 0x27f97 hash_collisions = 0x1651b43 hash_chains = 0x7ac3 hash_chain_max = 0x12 no_grow = 0x1 } Live kernel:> arc::print{ anon = ARC_anon mru = ARC_mru mru_ghost = ARC_mru_ghost mfu = ARC_mfu mfu_ghost = ARC_mfu_ghost size = 0x1b279400 p = 0x1a1dcaa4 c = 0x1a1dcaa4 c_min = 0x4000000 c_max = 0x5e114800 hits = 0xef7c96 misses = 0x25efa8 deleted = 0x1db537 recycle_miss = 0xa6221 mutex_miss = 0x12b59 evict_skip = 0x70d62b hash_elements = 0xcda1 hash_elements_max = 0x1b589 hash_collisions = 0x18e58a hash_chains = 0x3d16 hash_chain_max = 0xf no_grow = 0x1 } Should I post ::kmem_cache and/or ::kmastat somewhere? It''s about 2*(20+30)kB.. /Tomas -- Tomas ?gren, stric@acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Robert Milkowski
2007-Jan-10 16:26 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory
Hello Tomas, Wednesday, January 3, 2007, 10:32:39 AM, you wrote: T?> Hello. T?> Having some hangs on a snv53 machine which is quite probably ZFS+NFS T?> related, since that''s all the machine do ;) T?> The machine is a 2x750MHz Blade1000 with 2GB ram, using a SysKonnect T?> 9821 GigE card (with their 8.19.1.3 skge driver) and two HP branded MPT T?> SCSI cards. Normal load is pretty much "read all you can" with misc T?> tarballs and isos since it''s an NFS backend to our caching http/ftp T?> cluster delivering free software to the world. T?> What happens is that the machine just stops responding.. it can respond T?> to ping for a while (while userland is non-responsive, including T?> console) but after a while, that stops too.. T?> Produced a panic to get a dump and tried ::memstat; T?> unterweser:/scratch/070103# mdb unix.0 vmcore.0 T?> Loading modules: [ unix krtld genunix specfs dtrace ufs scsi_vhci pcisch T?> ssd fcp fctl qlc md ip hook neti sctp arp usba s1394 nca lofs zfs random T?> sd nfs ptm cpc ]>> ::memstatT?> Page Summary Pages MB %Tot T?> ------------ ---------------- ---------------- ---- T?> Kernel 250919 1960 98% T?> Anon 888 6 0% T?> Exec and libs 247 1 0% T?> Page cache 38 0 0% T?> Free (cachelist) 405 3 0% T?> Free (freelist) 4370 34 2% T?> Total 256867 2006 T?> Physical 253028 1976 T?> That doesn''t seem too healthy to me.. probably something kernely eating T?> up everything and the machine is just swapping to death or something.. T?> A dump from live kernel with mdb -k after 1.5h uptime; T?> Page Summary Pages MB %Tot T?> ------------ ---------------- ---------------- ---- T?> Kernel 212310 1658 83% T?> Anon 11307 88 4% T?> Exec and libs 2418 18 1% T?> Page cache 18400 143 7% T?> Free (cachelist) 4383 34 2% T?> Free (freelist) 8049 62 3% T?> The tweaks I have are: T?> set ncsize = 500000 T?> set nfs:nrnode = 50 T?> set zfs:zil_disable=1 T?> set zfs:zfs_vdev_cache_bshift=14 T?> set zfs:zfs_vdev_cache_size=0 Comment out ncsize and reboot - should help. I had similar behavior with increased ncsize from default on ZFS+NFS. Mark suggested to comment it out and it just works since then. I was also told increasing ncsize with ZFS isn''t needed that much (if at all) - however I haven''t done any testing and I don''t know technical details how DNLC is used with ZFS to be sure. -- Best regards, Robert mailto:rmilkowski@task.gda.pl http://milek.blogspot.com _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Mark Maybee
2007-Jan-10 16:26 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory
Ah yes! Thank you Casper. I knew this looked familiar! :-) Yes, this is almost certainly what is happening here. The bug was introduced in build 51 and fixed in build 54. Casper.Dik@Sun.COM wrote:>> Hmmm, so there is lots of evictable cache here (mostly in the MFU >> part of the cache)... could you make your core file available? >> I would like to take a look at it. > > Isn''t this just like: > 6493923 nfsfind on ZFS filesystem quickly depletes memory in a 1GB system > > Which was introduced in b51(or 52) and fixed in snv_54. > > Casper_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Casper.Dik@sun.com
2007-Jan-10 16:26 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory
>Hmmm, so there is lots of evictable cache here (mostly in the MFU >part of the cache)... could you make your core file available? >I would like to take a look at it.Isn''t this just like: 6493923 nfsfind on ZFS filesystem quickly depletes memory in a 1GB system Which was introduced in b51(or 52) and fixed in snv_54. Casper _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Richard Elling
2007-Jan-10 16:26 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory
Tomas ?gren wrote:> df (GNU df) says there are ~850k inodes used, I''d like to keep those in > memory.. There is currently 1.8TB used on the filesystem.. The > probability of a cache hit in the user data cache is about 0% and the > probability that an rsync happens again shortly is about 100%..Also, buy more RAM. Sometimes, throwing hardware at a big problem is a good answer :-) -- richard _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Tomas Ögren
2007-Jan-10 16:26 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory
On 03 January, 2007 - Mark Maybee sent me these 5,0K bytes:> Tomas, > > There are a couple of things going on here: > > 1. There is a lot of fragmentation in your meta-data caches (znode, > dnode, dbuf, etc). This is burning up about 300MB of space in your > hung kernel. This is a known problem that we are currently working > on.Great!> 2. While the ARC has set its desired size down to c_min (64MB), its > actually still consuming ~800MB in the hung kernel. This is odd. > The bulk of this space is in the 32K and 64K data caches. Could > you print out the contents of ARC_anon, ARC_mru, ARC_mfu, ARC_mru_ghost, > and ARC_mfu_ghost?Like this?> ARC_anon::print{ list = { list_size = 0 list_offset = 0 list_head = { list_next = 0 list_prev = 0 } } lsize = 0 size = 0x19c000 hits = 0 mtx = { _opaque = [ 0 ] } }> ARC_mru::print{ list = { list_size = 0x90 list_offset = 0x70 list_head = { list_next = 0x30072a5b5f8 list_prev = 0x300758b6c70 } } lsize = 0x1f88200 size = 0x3e5c200 hits = 0x44c48ad mtx = { _opaque = [ 0 ] } }> ARC_mfu::print{ list = { list_size = 0x90 list_offset = 0x70 list_head = { list_next = 0x30099c7a730 list_prev = 0x300dc11fee0 } } lsize = 0x2f2e4400 size = 0x318a8400 hits = 0x466bbec mtx = { _opaque = [ 0 ] } }> ARC_mru_ghost::print{ list = { list_size = 0x90 list_offset = 0x70 list_head = { list_next = 0x300758b6eb0 list_prev = 0x300d65faa10 } } lsize = 0x97a3cc00 size = 0x97a3cc00 hits = 0xfa4a49 mtx = { _opaque = [ 0 ] } }> ARC_mfu_ghost::print{ list = { list_size = 0x90 list_offset = 0x70 list_head = { list_next = 0x3006c7c8ce0 list_prev = 0x300d65fa2c0 } } lsize = 0x879ddc00 size = 0x879ddc00 hits = 0x3b37c8 mtx = { _opaque = [ 0 ] } } /Tomas -- Tomas ?gren, stric@acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Mark Maybee
2007-Jan-10 16:26 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory
Tomas, There are a couple of things going on here: 1. There is a lot of fragmentation in your meta-data caches (znode, dnode, dbuf, etc). This is burning up about 300MB of space in your hung kernel. This is a known problem that we are currently working on. 2. While the ARC has set its desired size down to c_min (64MB), its actually still consuming ~800MB in the hung kernel. This is odd. The bulk of this space is in the 32K and 64K data caches. Could you print out the contents of ARC_anon, ARC_mru, ARC_mfu, ARC_mru_ghost, and ARC_mfu_ghost? -Mark Tomas ?gren wrote:> Hello. > > Having some hangs on a snv53 machine which is quite probably ZFS+NFS > related, since that''s all the machine do ;) > > The machine is a 2x750MHz Blade1000 with 2GB ram, using a SysKonnect > 9821 GigE card (with their 8.19.1.3 skge driver) and two HP branded MPT > SCSI cards. Normal load is pretty much "read all you can" with misc > tarballs and isos since it''s an NFS backend to our caching http/ftp > cluster delivering free software to the world. > > What happens is that the machine just stops responding.. it can respond > to ping for a while (while userland is non-responsive, including > console) but after a while, that stops too.. > > Produced a panic to get a dump and tried ::memstat; > unterweser:/scratch/070103# mdb unix.0 vmcore.0 > Loading modules: [ unix krtld genunix specfs dtrace ufs scsi_vhci pcisch > ssd fcp fctl qlc md ip hook neti sctp arp usba s1394 nca lofs zfs random > sd nfs ptm cpc ] >> ::memstat > Page Summary Pages MB %Tot > ------------ ---------------- ---------------- ---- > Kernel 250919 1960 98% > Anon 888 6 0% > Exec and libs 247 1 0% > Page cache 38 0 0% > Free (cachelist) 405 3 0% > Free (freelist) 4370 34 2% > > Total 256867 2006 > Physical 253028 1976 > > That doesn''t seem too healthy to me.. probably something kernely eating > up everything and the machine is just swapping to death or something.. > > A dump from live kernel with mdb -k after 1.5h uptime; > Page Summary Pages MB %Tot > ------------ ---------------- ---------------- ---- > Kernel 212310 1658 83% > Anon 11307 88 4% > Exec and libs 2418 18 1% > Page cache 18400 143 7% > Free (cachelist) 4383 34 2% > Free (freelist) 8049 62 3% > > > The tweaks I have are: > set ncsize = 500000 > set nfs:nrnode = 50 > set zfs:zil_disable=1 > set zfs:zfs_vdev_cache_bshift=14 > set zfs:zfs_vdev_cache_size=0 > > Which according to ::kmem_cache results in about: > 0000030002e30008 dmu_buf_impl_t 0000 000000 328 487728 > 0000030002e30288 dnode_t 0000 000000 640 453204 > 0000030002e30508 arc_buf_hdr_t 0000 000000 144 103544 > 0000030002e30788 arc_buf_t 0000 000000 40 36743 > 0000030002e30a08 zil_lwb_cache 0000 000000 200 0 > 0000030002e30c88 zfs_znode_cache 0000 000000 200 453200 > > but those buffers equal to about 550MB.. > > dnlc_nentries on the hung has gone down to 15000.. (where are the rest > of the ~450k-15k dnode/znodes hanging out?) > > Hung kernel: >> arc::print > { > anon = ARC_anon > mru = ARC_mru > mru_ghost = ARC_mru_ghost > mfu = ARC_mfu > mfu_ghost = ARC_mfu_ghost > size = 0x358a0600 > p = 0x4000000 > c = 0x4000000 > c_min = 0x4000000 > c_max = 0x5e114800 > hits = 0xbc860fd > misses = 0x2f296e1 > deleted = 0x1d88739 > recycle_miss = 0xf7f30c > mutex_miss = 0x24b13d > evict_skip = 0x21501d02 > hash_elements = 0x27f97 > hash_elements_max = 0x27f97 > hash_collisions = 0x1651b43 > hash_chains = 0x7ac3 > hash_chain_max = 0x12 > no_grow = 0x1 > } > > > Live kernel: >> arc::print > { > anon = ARC_anon > mru = ARC_mru > mru_ghost = ARC_mru_ghost > mfu = ARC_mfu > mfu_ghost = ARC_mfu_ghost > size = 0x1b279400 > p = 0x1a1dcaa4 > c = 0x1a1dcaa4 > c_min = 0x4000000 > c_max = 0x5e114800 > hits = 0xef7c96 > misses = 0x25efa8 > deleted = 0x1db537 > recycle_miss = 0xa6221 > mutex_miss = 0x12b59 > evict_skip = 0x70d62b > hash_elements = 0xcda1 > hash_elements_max = 0x1b589 > hash_collisions = 0x18e58a > hash_chains = 0x3d16 > hash_chain_max = 0xf > no_grow = 0x1 > } > > > Should I post ::kmem_cache and/or ::kmastat somewhere? It''s about > 2*(20+30)kB.. > > /Tomas_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Tomas Ögren
2007-Jan-10 16:26 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory
On 03 January, 2007 - Robert Milkowski sent me these 0,2K bytes:> Hello Tomas, > > > Give us output of ::kmastat on crashdump.Ok, attached. /Tomas -- Tomas ?gren, stric@acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se -------------- next part -------------- cache buf buf buf memory alloc alloc name size in use total in use succeed fail ------------------------- ------ ------ ------ --------- --------- ----- kmem_magazine_1 16 213 20828 335872 1864891 0 kmem_magazine_3 32 215 9652 311296 1104585 0 kmem_magazine_7 64 177 6858 442368 1340533 0 kmem_magazine_15 128 159 3717 483328 918384 0 kmem_magazine_31 256 12 124 32768 720533 0 kmem_magazine_47 384 20 168 65536 261878 0 kmem_magazine_63 512 4 75 40960 5849 0 kmem_magazine_95 768 8 80 65536 2808 0 kmem_magazine_143 1152 0 0 0 0 0 kmem_slab_cache 56 37745 61770 3489792 3328105 0 kmem_bufctl_cache 24 237168 281031 6791168 4541237 0 kmem_bufctl_audit_cache 128 0 0 0 0 0 kmem_va_8192 8192 113889 153248 1255407616 332666 0 kmem_va_16384 16384 5281 21360 349962240 1121413 0 kmem_va_24576 24576 69 160 4194304 16047 0 kmem_va_32768 32768 11567 15112 495190016 3019065 0 kmem_va_40960 40960 63 150 6553600 242545 0 kmem_va_49152 49152 6 25 1310720 3429 0 kmem_va_57344 57344 116 148 9699328 386019 0 kmem_va_65536 65536 6579 9456 619708416 1201954 0 kmem_alloc_8 8 6517 15255 122880 14368906 0 kmem_alloc_16 16 7735 9652 155648 299158744 0 kmem_alloc_24 24 5107 7119 172032 158911294 0 kmem_alloc_32 32 170388 182880 5898240 592275780 0 kmem_alloc_40 40 26644 66178 2670592 53208818 0 kmem_alloc_48 48 4095 25181 1220608 63957055 0 kmem_alloc_56 56 3989 36250 2048000 7803335 0 kmem_alloc_64 64 5967 135255 8724480 47214667 0 kmem_alloc_80 80 37731 547218 44384256 36492540 0 kmem_alloc_96 96 26467 225624 22003712 10696361 0 kmem_alloc_112 112 70229 198720 22609920 99166862 0 kmem_alloc_128 128 1332 18207 2367488 7348927 0 kmem_alloc_160 160 1530 5450 892928 2267112 0 kmem_alloc_192 192 251 378 73728 1977811 0 kmem_alloc_224 224 179 216 49152 1371127 0 kmem_alloc_256 256 4687 21359 5644288 168546954 0 kmem_alloc_320 320 550 600 196608 2856787 0 kmem_alloc_384 384 394 588 229376 50090330 0 kmem_alloc_448 448 1644 3780 1720320 467081636 0 kmem_alloc_512 512 290 600 327680 20124863 0 kmem_alloc_640 640 1299 2112 1441792 435757788 0 kmem_alloc_768 768 18 30 24576 198711 0 kmem_alloc_896 896 9 18 16384 244765 0 kmem_alloc_1152 1152 701 742 868352 1924822 0 kmem_alloc_1344 1344 166 168 229376 471847 0 kmem_alloc_1600 1600 23 35 57344 3621755 0 kmem_alloc_2048 2048 83 100 204800 3590335 0 kmem_alloc_2688 2688 128 138 376832 833976 0 kmem_alloc_4096 4096 41 46 188416 159283 0 kmem_alloc_8192 8192 849 852 6979584 191304 0 kmem_alloc_12288 12288 9 16 196608 98934 0 kmem_alloc_16384 16384 37 37 606208 228 0 streams_mblk 64 7792 23876 1540096 589535033 0 streams_dblk_16 128 49 63 8192 409717 0 streams_dblk_80 192 744 756 147456 733832 124 streams_dblk_144 256 3 62 16384 563294544 0 streams_dblk_208 320 241 350 114688 106806 0 streams_dblk_272 384 7 21 8192 50801 0 streams_dblk_336 448 0 18 8192 790 0 streams_dblk_528 640 0 12 8192 5723 0 streams_dblk_1040 1152 1 21 24576 4808 0 streams_dblk_1488 1600 0 10 16384 2727 0 streams_dblk_1936 2048 6230 6288 12877824 431230297 16957 streams_dblk_2576 2688 0 6 16384 7259 0 streams_dblk_3920 4032 0 4 16384 4245 0 streams_dblk_8192 112 0 63 8192 132105 0 streams_dblk_12112 12224 0 4 49152 10915 0 streams_dblk_16384 112 0 63 8192 12915 0 streams_dblk_20304 20416 0 2 40960 8644 0 streams_dblk_24576 112 0 63 8192 16343 0 streams_dblk_28496 28608 0 4 114688 7807 0 streams_dblk_32768 112 18 126 16384 24252395 0 streams_dblk_36688 36800 0 6 221184 52845 0 streams_dblk_40960 112 0 63 8192 710 0 streams_dblk_44880 44992 0 4 180224 845 0 streams_dblk_49152 112 0 63 8192 547 0 streams_dblk_53072 53184 0 4 212992 644 0 streams_dblk_57344 112 0 63 8192 519 0 streams_dblk_61264 61376 0 2 122880 669 0 streams_dblk_65536 112 0 63 8192 456 0 streams_dblk_69456 69568 0 4 278528 423 0 streams_dblk_73728 112 0 63 8192 340 0 streams_dblk_esb 112 89 126 16384 24676 0 streams_fthdr 264 0 0 0 0 0 streams_ftblk 232 0 0 0 0 0 multidata 248 0 0 0 0 0 multidata_pdslab 7112 0 0 0 0 0 multidata_pattbl 32 0 0 0 0 0 log_cons_cache 48 9 169 8192 51 0 taskq_ent_cache 56 2635 4060 229376 8701976 0 taskq_cache 216 66 74 16384 88 0 id32_cache 32 0 0 0 0 0 bp_map_16384 16384 0 32 524288 20 0 bp_map_32768 32768 0 16 524288 128 0 bp_map_49152 49152 0 10 524288 121 0 bp_map_65536 65536 0 8 524288 73 0 bp_map_81920 81920 0 6 524288 45 0 bp_map_98304 98304 0 5 524288 39 0 bp_map_114688 114688 0 4 524288 24 0 bp_map_131072 131072 0 4 524288 12 0 memseg_cache 112 0 0 0 0 0 mod_hash_entries 24 322 678 16384 2944 0 ipp_mod 304 0 0 0 0 0 ipp_action 368 0 0 0 0 0 ipp_packet 64 0 0 0 0 0 mmuctxdom_cache 192 2 42 8192 2 0 sfmmuid_cache 184 44 84 16384 2187 0 sfmmu_tsbinfo_cache 64 43 127 8192 4554 0 sfmmu_tsb8k_cache 8192 0 0 0 0 0 sfmmu_tsb_cache 8192 22 22 180224 2599 0 sfmmu8_cache 312 43709 72696 22904832 3766079 0 sfmmu1_cache 88 163 1840 163840 19088 0 pa_hment_cache 64 30 127 8192 2377 0 ism_blk_cache 272 0 0 0 0 0 ism_ment_cache 32 0 0 0 0 0 seg_cache 72 2262 2599 188416 95934 0 dev_info_node_cache 488 174 256 131072 667 0 segkp_8192 8192 36 48 393216 1231 0 segkp_16384 16384 0 0 0 0 0 segkp_24576 24576 0 0 0 0 0 segkp_32768 32768 652 688 22544384 25331 0 segkp_40960 40960 0 0 0 0 0 umem_np_8192 8192 0 32 262144 212 0 umem_np_16384 16384 0 16 262144 96 0 umem_np_24576 24576 0 0 0 0 0 umem_np_32768 32768 0 8 262144 4 0 umem_np_40960 40960 0 0 0 0 0 umem_np_49152 49152 0 0 0 0 0 umem_np_57344 57344 0 0 0 0 0 umem_np_65536 65536 0 4 262144 4 0 thread_cache 800 351 360 294912 15488 0 wbuf32_cache 512 349 360 196608 15483 0 wbuf64_cache 1024 2 14 16384 17 0 lwp_cache 912 351 352 360448 1142 0 turnstile_cache 64 643 762 49152 33314 0 tslabel_cache 48 2 169 8192 2 0 cred_cache 172 229 276 49152 16352 0 rctl_cache 40 675 1015 40960 23103 0 rctl_val_cache 64 1318 1778 114688 47167 0 task_cache 104 31 78 8192 399 0 cyclic_id_cache 64 10 127 8192 10 0 dnlc_space_cache 24 0 339 8192 158 0 vn_cache 240 68620 496186 131121152 655784 0 vsk_anchor_cache 40 13 203 8192 24 0 file_cache 56 418 580 32768 169501 0 stream_head_cache 368 211 242 90112 31176 0 queue_cache 656 495 564 385024 107675 0 syncq_cache 160 9 50 8192 27 0 qband_cache 64 0 0 0 0 0 linkinfo_cache 48 4 169 8192 4 0 ciputctrl_cache 128 4 63 8192 4 0 serializer_cache 64 46 127 8192 126 0 as_cache 224 43 72 16384 2186 0 marker_cache 128 0 63 8192 3636 0 anon_cache 48 13595 13689 663552 109919 0 anonmap_cache 64 1488 1778 114688 45757 0 segvn_cache 104 2262 2574 270336 89156 0 flk_edges 48 0 169 8192 2 0 fdb_cache 104 0 0 0 0 0 timer_cache 136 1 59 8192 2 0 physio_buf_cache 248 0 32 8192 282 0 snode_cache 152 317 371 57344 142080 0 ufs_inode_cache 368 4318 4334 1613824 4771 0 directio_buf_cache 272 0 0 0 0 0 lufs_save 24 0 339 8192 2272 0 lufs_bufs 256 0 31 8192 2333 0 lufs_mapentry_cache 112 0 72 8192 33595 0 pcisch0_dvma_8192 8192 40 112 917504 239544936 0 pcisch1_dvma_8192 8192 1282 1304 10682368 466966312 0 fctl_cache 112 0 72 8192 7 0 fp1_cache 728 1 11 8192 6 0 fcp1_cache 1224 2 13 16384 148681 0 dv_node_cache 128 62 252 32768 1170 0 sdev_node_cache 200 790 800 163840 848 0 clnt_clts_endpnt_cache 88 0 92 8192 1 0 md_stripe_parent 96 0 84 8192 100979 0 md_stripe_child 312 0 26 8192 113462 0 md_mirror_parent 160 0 50 8192 70212 0 md_mirror_child 304 0 26 8192 113463 0 md_mirror_wow 16440 0 8 139264 635 0 kcf_sreq_cache 48 0 0 0 0 0 kcf_areq_cache 272 0 0 0 0 0 kcf_context_cache 88 0 0 0 0 0 ipsec_actions 72 0 113 8192 1327 0 ipsec_selectors 80 0 0 0 0 0 ipsec_policy 72 0 0 0 0 0 ipsec_info 336 0 24 8192 2100 0 ip_minor_arena_1 1 60 128 128 29183 0 ipcl_conn_cache 488 36 45 24576 9461 0 ipcl_tcpconn_cache 1696 30 54 98304 40222 0 radix_mask 32 1 254 8192 1 0 radix_node 112 2 72 8192 2 0 rt_entry 160 3 50 8192 4 0 ire_cache 344 31 69 24576 1916 0 tcp_timercache 88 55 184 16384 117695 0 tcp_sack_info_cache 80 5 101 8192 19109 0 tcp_iphc_cache 120 29 67 8192 39541 0 squeue_cache 136 2 42 8192 2 0 sctp_conn_cache 2264 1 7 16384 1 0 sctp_faddr_cache 168 0 0 0 0 0 sctp_set_cache 24 0 0 0 0 0 sctp_ftsn_set_cache 16 0 0 0 0 0 ire_gw_secattr_cache 32 0 0 0 0 0 sctpsock 616 0 0 0 0 0 sctp_assoc 64 0 0 0 0 0 socktpi_cache 456 15 34 16384 2521 0 socktpi_unix_cache 456 50 68 32768 99 0 mac_impl_cache 376 0 0 0 0 0 dls_cache 152 0 0 0 0 0 soft_ring_cache 176 0 0 0 0 0 dls_vlan_cache 48 0 0 0 0 0 dls_link_cache 368 0 0 0 0 0 dld_ctl_1 1 0 0 0 0 0 dld_str_cache 272 0 29 8192 1 0 udp_cache 416 31 54 24576 9447 0 process_cache 3080 45 60 196608 1124 0 exacct_object_cache 40 0 0 0 0 0 ch_private_cache 13200 2 2 32768 2 0 hal0_cache 464 0 17 8192 1 0 kssl_cache 1560 0 0 0 0 0 tl_cache 432 77 90 40960 366 0 keysock_1 1 0 0 0 0 0 spdsock_1 1 0 64 64 1 0 fnode_cache 176 23 42 8192 3344 0 pipe_cache 320 17 50 16384 372 0 namefs_inodes_1 1 17 64 64 20 0 port_cache 80 3 101 8192 3 0 ip_minor_1 1 0 0 0 0 0 ar_minor_1 1 0 0 0 0 0 lnode_cache 32 2 254 8192 2 0 zio_buf_512 512 63141 156405 85417984 19564665 0 zio_buf_1024 1024 605 936 958464 4314404 0 zio_buf_1536 1536 198 270 442368 4611683 0 zio_buf_2048 2048 226 296 606208 676617 0 zio_buf_2560 2560 75 105 286720 306878 0 zio_buf_3072 3072 182 280 860160 538528 0 zio_buf_3584 3584 49 81 294912 354593 0 zio_buf_4096 4096 50 66 270336 275513 0 zio_buf_5120 5120 42 104 532480 1114734 0 zio_buf_6144 6144 37 48 294912 439142 0 zio_buf_7168 7168 317 416 2981888 83940957 0 zio_buf_8192 8192 26 29 237568 97297 0 zio_buf_10240 10240 14 28 286720 87990 0 zio_buf_12288 12288 10 14 172032 36740 0 zio_buf_14336 14336 77 140 2007040 18157638 0 zio_buf_16384 16384 5214 5224 85590016 4526582 0 zio_buf_20480 20480 23 28 573440 7541438 0 zio_buf_24576 24576 3 5 122880 10210 0 zio_buf_28672 28672 30 30 860160 8440188 0 zio_buf_32768 32768 11554 11556 378667008 20610854 0 zio_buf_40960 40960 27 27 1105920 2581957 0 zio_buf_49152 49152 3 6 294912 65294 0 zio_buf_57344 57344 3 6 344064 59908 0 zio_buf_65536 65536 6575 6579 431161344 5630613 0 zio_buf_73728 73728 0 2 147456 61554 0 zio_buf_81920 81920 0 2 163840 34262 0 zio_buf_90112 90112 0 2 180224 31718 0 zio_buf_98304 98304 0 4 393216 32555 0 zio_buf_106496 106496 0 2 212992 48917 0 zio_buf_114688 114688 0 4 458752 23059 0 zio_buf_122880 122880 0 3 368640 23344 0 zio_buf_131072 131072 0 4 524288 606961 0 dmu_buf_impl_t 328 87267 515304 175890432 43706644 0 dnode_t 640 62641 379824 259293184 11064793 0 arc_buf_hdr_t 144 163735 163744 23953408 32515851 0 arc_buf_t 40 25387 30856 1245184 53461222 0 zil_lwb_cache 200 0 0 0 0 0 zfs_znode_cache 200 62605 493480 101064704 11075780 0 mpt1_cache 536 7 150 81920 85012498 0 mpt0_cache 536 7 90 49152 85896363 0 md_raid_parent 120 0 0 0 0 0 md_raid_child 1040 0 0 0 0 0 md_raid_cbufs 376 0 0 0 0 0 md_trans_parent 80 0 0 0 0 0 md_trans_child 248 0 0 0 0 0 icmp_minor_1 1 0 0 0 0 0 glm0_cache 472 4 17 8192 1196 0 authkern_cache 72 0 113 8192 77 0 authloopback_cache 72 0 0 0 0 0 authdes_cache_handle 80 0 0 0 0 0 rnode_cache 648 0 12 8192 16 0 nfs_access_cache 56 0 145 8192 16 0 client_handle_cache 32 0 254 8192 1 0 rnode4_cache 960 0 0 0 0 0 svnode_cache 40 0 0 0 0 0 nfs4_access_cache 56 0 0 0 0 0 client_handle4_cache 32 0 0 0 0 0 nfs4_ace4vals_cache 48 0 0 0 0 0 nfs4_ace4_list_cache 264 0 0 0 0 0 NFS_idmap_cache 48 0 0 0 0 0 nfslog_small_rec 512 0 0 0 0 0 nfslog_medium_rec 8192 0 0 0 0 0 nfslog_large_rec 32768 0 0 0 0 0 exi_cache_handle 40 0 203 8192 44 0 dtrace_state_cache 2048 0 0 0 0 0 lm_vnode 184 0 44 8192 8 0 lm_xprt 32 2 254 8192 2 0 lm_sysid 160 1 50 8192 2 0 lm_client 128 0 63 8192 4 0 lm_async 32 0 0 0 0 0 lm_sleep 96 0 0 0 0 0 lm_config 80 2 101 8192 15 0 glm1_cache 472 0 17 8192 38 0 pty_map 64 1 127 8192 1 0 Client_entry_cache 744 0 0 0 0 0 OpenOwner_entry_cache 528 0 0 0 0 0 OpenStateID_entry_cache 256 0 0 0 0 0 LockStateID_entry_cache 504 0 0 0 0 0 Lockowner_entry_cache 176 0 0 0 0 0 File_entry_cache 288 0 0 0 0 0 DelegStateID_entry_cache 248 0 0 0 0 0 md_softpart_parent 88 0 0 0 0 0 md_softpart_child 304 0 0 0 0 0 ------------------------- ------ ------ ------ --------- --------- ----- Total [static] 565248 35554 0 Total [hat_memload] 22904832 3766079 0 Total [kmem_msb] 12058624 14088803 0 Total [kmem_va] 2742026240 6323138 0 Total [kmem_default] 1844068352 385664405 17081 Total [bp_map] 4194304 462 0 Total [kmem_tsb_default] 180224 2599 0 Total [hat_memload1] 163840 19088 0 Total [umem_np] 1048576 316 0 Total [segkp] 22937600 26562 0 Total [pcisch0_dvma] 917504 239544936 0 Total [pcisch1_dvma] 10682368 466966312 0 Total [ip_minor_arena] 128 29183 0 Total [spdsock] 64 1 0 Total [namefs_inodes] 64 20 0 ------------------------- ------ ------ ------ --------- --------- ----- vmem memory memory memory alloc alloc name in use total import succeed fail ------------------------- --------- ---------- --------- --------- ----- heap 2997354496 4398046511104 0 69116 0 vmem_metadata 25698304 25952256 25952256 3052 0 vmem_seg 24494080 24494080 24494080 2990 0 vmem_hash 893568 901120 901120 62 0 vmem_vmem 278400 337904 303104 88 0 static 589824 589824 589824 70 0 static_alloc 8320 16384 16384 3 0 hat_memload 22904832 22904832 22904832 2800 0 kstat 253112 270336 204800 845 0 kmem_metadata 14098432 31719424 31719424 5687 0 kmem_msb 12058624 12058624 12058624 5214 0 kmem_cache 197816 221184 221184 372 0 kmem_hash 1806848 1818624 1818624 1574 140 kmem_log 131488 139264 139264 6 0 kmem_firewall_va 145424384 145424384 145424384 21199 0 kmem_firewall 0 0 0 0 0 kmem_oversize 123254125 123404288 123404288 21184 0 mod_sysfile 892 8192 8192 25 0 kmem_va 2745630720 2745630720 2745630720 44364 0 kmem_default 1844068352 1844068352 1844068352 3322142 17081 little_endian 179168 204800 204800 307 0 big_endian 865592 1130496 1130496 4012 0 bp_map 4194304 4194304 4194304 16 0 ksyms 1584856 1753088 1753088 258 0 ctf 1052532 1196032 1196032 259 0 kmem_tsb 4194304 4194304 4194304 1 0 kmem_tsb_default 475136 4194304 4194304 152 0 hat_memload1 163840 163840 163840 29 0 umem_np 1048576 1048576 1048576 7 0 heap32 2234112 134217728 0 503 0 id32 0 0 0 0 0 module_data 1552240 2441216 2179072 345 0 promplat 0 0 0 460 0 heaptext 38682624 134217728 0 110 0 module_text 6458180 7176192 5120000 255 0 logminor_space 25 262137 0 67 0 taskq_id_arena 30 2147483647 0 52 0 segkp 22937600 2147483648 0 2120 0 rctl_ids 32 32767 0 32 0 zoneid_space 0 9998 0 0 0 taskid_space 31 999999 0 399 0 pool_ids 0 999998 0 0 0 contracts 37 2147483646 0 425 0 regspec 753664 5368709120 0 121 0 pcisch0_dvma 917504 1040187392 0 572406 0 pcisch1_dvma 11509760 1040187392 0 22464 0 ip_minor_arena 128 262140 0 2 0 dld_ctl 0 4294967295 0 0 0 dld_minor_arena 1 4294967295 0 1 0 tl_minor_space 77 262138 0 342 0 keysock 0 4294967295 0 0 0 spdsock 64 4294967295 0 1 0 namefs_inodes 64 65536 0 1 0 ip_minor 0 262142 0 0 0 ar_minor 0 262142 0 0 0 devfsadm_event_channel 1 101 0 1 0 devfsadm_event_channel 1 2 0 1 0 syseventconfd_event_channel 0 101 0 0 0 syseventconfd_event_channel 1 2 0 1 0 syseventd_channel 2 101 0 2 0 syseventd_channel 1 2 0 1 0 icmp_minor 0 262142 0 0 0 lmsysid_space 2 16383 0 3 0 dtrace 10 4294967295 0 2482 0 dtrace_minor 0 4294967293 0 0 0 ptms_minor 1 16 0 1 0 Client_id_space 0 131072 0 0 0 OpenOwner_id_space 0 1048576 0 0 0 OpenStateID_id_space 0 1048576 0 0 0 LockStateID_id_space 0 1048576 0 0 0 Lockowner_id_space 0 1048576 0 0 0 DelegStateID_id_space 0 1048576 0 0 0 heaptext_holesrc_15 8192 2097152 0 5 0 heaptext_hole_15 2852 8192 8192 10 0 heaptext_holesrc_14 49152 2097152 0 4 0 heaptext_hole_14 47628 49152 49152 6 0 module_text_holesrc 0 4194304 0 2 0 heaptext_hole_0 0 0 0 2 0 heaptext_holesrc_16 8192 2097152 0 2 0 heaptext_hole_16 2356 8192 8192 7 0 ------------------------- --------- ---------- --------- --------- ----- -------------- next part -------------- _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Robert Milkowski
2007-Jan-10 16:26 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory
Hello Tomas,
Give us output of ::kmastat on crashdump.
--
Best regards,
Robert mailto:rmilkowski@task.gda.pl
http://milek.blogspot.com
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Tomas Ögren
2007-Jan-10 16:26 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory
On 03 January, 2007 - Robert Milkowski sent me these 3,0K bytes:> Hello Tomas, > > Wednesday, January 3, 2007, 10:32:39 AM, you wrote: > > T?> The tweaks I have are: > T?> set ncsize = 500000 > T?> set nfs:nrnode = 50 > T?> set zfs:zil_disable=1 > T?> set zfs:zfs_vdev_cache_bshift=14 > T?> set zfs:zfs_vdev_cache_size=0 > > Comment out ncsize and reboot - should help. > I had similar behavior with increased ncsize from default on ZFS+NFS. > Mark suggested to comment it out and it just works since then. > > I was also told increasing ncsize with ZFS isn''t needed that much (if > at all) - however I haven''t done any testing and I don''t know > technical details how DNLC is used with ZFS to be sure.The thing is, I want DNLC-like cache Very Much.. There are rsync''s running against large trees all the time and if we can save on disk io there, it''s a Big Win.. df (GNU df) says there are ~850k inodes used, I''d like to keep those in memory.. There is currently 1.8TB used on the filesystem.. The probability of a cache hit in the user data cache is about 0% and the probability that an rsync happens again shortly is about 100%.. So I''d like to get rid of just about all data caching and just do metadata caching instead.. That''s why I turned off the vdev cache for instance, cache is evicted before it''s used again.. According to Sanjeev Bagewadi @sun.com:>But in the ZFS world, DNLC is part of the ARC, right? > >Not really... ZFS uses the regular DNLC for lookup optimization. However, the metadata/data is cached in the ARC. --snip-- It''s interesting though that dnlc_nentries says it''s lowered to its minimum @ 15k entires and arc::print has lowered it''s "c" to "c_min" (which is 64MB).. So memory is definately going elsewhere.. I can try lowering ncsize, but that is just about the opposite of what I want. I tried changing arc_reduce_dnlc_percent=0 to keep my dnlc cache and let someone else free their memory instead, but that lead to death much faster.. Adding more memory seem to just give you more time, not solve the problem.. /Tomas -- Tomas ?gren, stric@acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Mark Maybee
2007-Jan-10 16:26 UTC
[zfs-discuss] ZFS related (probably) hangs due to memory
Hmmm, so there is lots of evictable cache here (mostly in the MFU part of the cache)... could you make your core file available? I would like to take a look at it. -Mark Tomas ?gren wrote:> On 03 January, 2007 - Mark Maybee sent me these 5,0K bytes: > >> Tomas, >> >> There are a couple of things going on here: >> >> 1. There is a lot of fragmentation in your meta-data caches (znode, >> dnode, dbuf, etc). This is burning up about 300MB of space in your >> hung kernel. This is a known problem that we are currently working >> on. > > Great! > >> 2. While the ARC has set its desired size down to c_min (64MB), its >> actually still consuming ~800MB in the hung kernel. This is odd. >> The bulk of this space is in the 32K and 64K data caches. Could >> you print out the contents of ARC_anon, ARC_mru, ARC_mfu, ARC_mru_ghost, >> and ARC_mfu_ghost? > > Like this? > >> ARC_anon::print > { > list = { > list_size = 0 > list_offset = 0 > list_head = { > list_next = 0 > list_prev = 0 > } > } > lsize = 0 > size = 0x19c000 > hits = 0 > mtx = { > _opaque = [ 0 ] > } > } >> ARC_mru::print > { > list = { > list_size = 0x90 > list_offset = 0x70 > list_head = { > list_next = 0x30072a5b5f8 > list_prev = 0x300758b6c70 > } > } > lsize = 0x1f88200 > size = 0x3e5c200 > hits = 0x44c48ad > mtx = { > _opaque = [ 0 ] > } > } >> ARC_mfu::print > { > list = { > list_size = 0x90 > list_offset = 0x70 > list_head = { > list_next = 0x30099c7a730 > list_prev = 0x300dc11fee0 > } > } > lsize = 0x2f2e4400 > size = 0x318a8400 > hits = 0x466bbec > mtx = { > _opaque = [ 0 ] > } > } >> ARC_mru_ghost::print > { > list = { > list_size = 0x90 > list_offset = 0x70 > list_head = { > list_next = 0x300758b6eb0 > list_prev = 0x300d65faa10 > } > } > lsize = 0x97a3cc00 > size = 0x97a3cc00 > hits = 0xfa4a49 > mtx = { > _opaque = [ 0 ] > } > } >> ARC_mfu_ghost::print > { > list = { > list_size = 0x90 > list_offset = 0x70 > list_head = { > list_next = 0x3006c7c8ce0 > list_prev = 0x300d65fa2c0 > } > } > lsize = 0x879ddc00 > size = 0x879ddc00 > hits = 0x3b37c8 > mtx = { > _opaque = [ 0 ] > } > } > > /Tomas_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss