Hi, I'm trying to tar a rather big directory via nfs (some 800gb), it has many subdirectories, some of them with many files (close to 10^6 :-) just before the server panics, the tar (on the client) starts complaining about lost files, or permition denied, but not in the pathological directories. panic: kmem_malloc(-1661382656): kmem_map too small: 645009408 total allocated cpuid = 3 KDB: enter: panic [thread pid 881 tid 100112 ] Stopped at kdb_enter_why+0x3d: movq $0,0x5ef3e8(%rip) db> tr Tracing pid 881 tid 100112 td 0xffffff0004ba2000 kdb_enter_why() at kdb_enter_why+0x3d panic() at panic+0x17b kmem_malloc() at kmem_malloc+0x565 uma_large_malloc() at uma_large_malloc+0x4a malloc() at malloc+0xd7 nfsrv_readdir() at nfsrv_readdir+0x4e1 nfssvc() at nfssvc+0x400 syscall() at syscall+0x1bb Xfast_syscall() at Xfast_syscall+0xab --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x8006885cc, rsp = 0x7fffffffea28, rbp = 0 --- I have increased vm.kmem_size_max="1024M" vm.kmem_size="1024M" vfs.zfs.arc_max="800M" it just seems to delay the panic though, it smells like some memory leak ... the host is running amd64 quad core, 7.1-prerelease and 8GB. danny
Danny Braniss wrote:> Hi, > I'm trying to tar a rather big directory via nfs (some 800gb), it > has many subdirectories, some of them with many files (close to 10^6 :-) > > just before the server panics, the tar (on the client) starts complaining > about lost > files, or permition denied, but not in the pathological directories. > panic: kmem_malloc(-1661382656): kmem_map too small: 645009408 total allocated > cpuid = 3 > KDB: enter: panic > [thread pid 881 tid 100112 ] > Stopped at kdb_enter_why+0x3d: movq $0,0x5ef3e8(%rip) > db> tr > Tracing pid 881 tid 100112 td 0xffffff0004ba2000 > kdb_enter_why() at kdb_enter_why+0x3d > panic() at panic+0x17b > kmem_malloc() at kmem_malloc+0x565 > uma_large_malloc() at uma_large_malloc+0x4a > malloc() at malloc+0xd7 > nfsrv_readdir() at nfsrv_readdir+0x4e1 > nfssvc() at nfssvc+0x400 > syscall() at syscall+0x1bb > Xfast_syscall() at Xfast_syscall+0xab > --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x8006885cc, rsp = > 0x7fffffffea28, rbp = 0 --- > > I have increased > vm.kmem_size_max="1024M" > vm.kmem_size="1024M" > vfs.zfs.arc_max="800M" > it just seems to delay the panic though, it smells like some memory leak ...Well, the canonical fix seems be to DECREASE vfs.zfs.arc_max to something like 100M and keep decreasing until it works. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20081216/5054df09/signature.pgp
>> it just seems to delay the panic though, it smells like some memory leak ... >> > > Well, the canonical fix seems be to DECREASE vfs.zfs.arc_max to > something like 100M and keep decreasing until it works. >More info here: http://wiki.freebsd.org/ZFSQuickStartGuide Once you tune, your problems will go away. The default install should not need tuning (so people stop posting this panic problem)... will that be fixed in the next stable release? Rudy