pluknet
2009-Aug-05 05:38 UTC
softdep_setup_freeblocks: kmem_malloc(4096): kmem_map too small
Hi.
We have a problem with user running with exceed quota:
Disk quotas for user eviluser (uid 9181):
Filesystem usage quota limit grace files quota limit grace
/home 6172656 6172672 6172672 14723 0 0
Some types of ufs operations running under him lead to kernel
panic due to out of kernel memory (tested on 6.2-R, and 6.4-R):
db> x/s *panicstr
buf.1: kmem_malloc(4096): kmem_map too small: 335544320 total allocated
Upping the higher level of vm.kmem_size_max doesn't help much,
postponing that panic little farther.
db> bt
Tracing pid 7242 tid 100772 td 0xca7a57d0
kdb_enter(c0924e28) at kdb_enter+0x2b
panic(c093a575,1000,14000000,c17f7818,0,...) at panic+0x127
kmem_malloc(c10680c0,1000,402,ef34e7bc,c07fb86d,...) at kmem_malloc+0x7d
page_alloc(c10613c0,1000,ef34e7af,402,0,...) at page_alloc+0x1a
slab_zalloc(c10613c0,402,c1061480,c10613c0,da68220c,...) at slab_zalloc+0xdd
uma_zone_slab(c10613c0,502) at uma_zone_slab+0xe8
uma_zalloc_bucket(c10613c0,502) at uma_zalloc_bucket+0x15c
uma_zalloc_arg(c10613c0,0,502) at uma_zalloc_arg+0x292
malloc(b8,c09d4ba0,502,0,0,...) at malloc+0x46
softdep_setup_freeblocks(cc8fb18c,0,0,800,cc8fb18c,ffffffe0,ffffffff,0,0) at sof
tdep_setup_freeblocks+0x48
ffs_truncate(c89e3990,0,0,800,c94f4300,...) at ffs_truncate+0x5cb
ffs_write(ef34ebec) at ffs_write+0x603
VOP_WRITE_APV(c09d5960,ef34ebec) at VOP_WRITE_APV+0xce
vn_write(caba7000,ef34ecbc,c94f4300,0,ca7a57d0) at vn_write+0x1ee
dofilewrite(ca7a57d0,7,caba7000,ef34ecbc,ffffffff,...) at dofilewrite+0x77
kern_writev(ca7a57d0,7,ef34ecbc,7e99c3c,42f6e8,...) at kern_writev+0x3b
write(ca7a57d0,ef34ed04) at write+0x45
syscall(3b,82b003b,bfbf003b,8851,82bb000,...) at syscall+0x2bf
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (4, FreeBSD ELF32, write), eip = 0x281ae32f, esp = 0xbfbfbbdc, ebp
0xbfbfbbf8 ---
Always, afaics, the source of panics is a process running apache:
db> show proc 7242
Process 7242 (httpd) at 0xca7a3648:
state: NORMAL
uid: 9181 gids: 999, 999, 9181
parent: pid 3799 at 0xca468000
ABI: FreeBSD ELF32
arguments: /home/eviluser/etc/apache/bin/httpd
threads: 1
100772 Run CPU 5 httpd
The type of a ufs operations is always the same:
a process of that user tries to write data to fs and falls back
(due to exceed quotas) to ffs_truncate() where it panics.
I couldn't reproduce it, that happens once per several days.
freeblks malloc type looks a bit leaky and a cause of panic.
db> show malloc
Type Allocs Frees Used
...
freeblks 3233513 2578006 655507
...
db> show lockedbufs
buf at 0xdbd84c88
b_flags = 0x20000220<vmio,done,cache>
b_error = 0, b_bufsize = 16384, b_bcount = 16384, b_resid = 0
b_bufobj = (0xcc1bfa50), b_data = 0xdc855000, b_blkno = 1624763616
b_npages = 4, pages(OBJ, IDX, PA): (0xcdb2cce4, 0xc, 0xb638b000),(0xcdb2cce4, 0x
d, 0x6fe2c000),(0xcdb2cce4, 0xe, 0xca8d000),(0xcdb2cce4, 0xf, 0xb216e000)
lock type bufwait: EXCL (count 1) by thread 0xcb41e190 (pid 11437)
buf at 0xdbd86398
b_flags = 0x20000000<vmio>
b_error = 0, b_bufsize = 16384, b_bcount = 16384, b_resid = 0
b_bufobj = (0xc8165c70), b_data = 0xdc89d000, b_blkno = 545231232
b_npages = 4, pages(OBJ, IDX, PA): (0xc81695ac, 0x40ff230, 0x842dc000),(0xc81695
ac, 0x40ff231, 0xdfd000),(0xc81695ac, 0x40ff232, 0x3b3e000),(0xc81695ac, 0x40ff2
33, 0x201f000)
db> show lockedvnods
Locked vnodes
0xcc144550: tag ufs, type VDIR
usecount 1, writecount 0, refcount 3 mountedhere 0
flags ()
v_object 0xc9c6e000 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xd00e4af0 (pid 11041)
ino 203063289, on dev aacd0s1g
0xcc1bf990: tag ufs, type VDIR
usecount 1, writecount 0, refcount 6 mountedhere 0
flags ()
v_object 0xcdb2cce4 ref 0 pages 22
lock type ufs: EXCL (count 1) by thread 0xcb41e190 (pid 11437)
with 1 pending
ino 203063639, on dev aacd0s1g
0xcdd84dd0: tag ufs, type VDIR
usecount 1, writecount 0, refcount 3 mountedhere 0
flags ()
lock type ufs: EXCL (count 1) by thread 0xc8de8000 (pid 3612)
ino 263490091, on dev aacd0s1g
0xc89e3990: tag ufs, type VREG
usecount 1, writecount 1, refcount 3 mountedhere 0
flags ()
v_object 0xc98a1294 ref 0 pages 4
lock type ufs: EXCL (count 1) by thread 0xca7a57d0 (pid 7242)
ino 140245401, on dev aacd0s1g
--
wbr,
pluknet
Kostik Belousov
2009-Aug-05 15:56 UTC
softdep_setup_freeblocks: kmem_malloc(4096): kmem_map too small
On Wed, Aug 05, 2009 at 09:38:13AM +0400, pluknet wrote:> Hi. > > We have a problem with user running with exceed quota: > > Disk quotas for user eviluser (uid 9181): > Filesystem usage quota limit grace files quota limit grace > /home 6172656 6172672 6172672 14723 0 0 > > Some types of ufs operations running under him lead to kernel > panic due to out of kernel memory (tested on 6.2-R, and 6.4-R): > > db> x/s *panicstr > buf.1: kmem_malloc(4096): kmem_map too small: 335544320 total allocated > > Upping the higher level of vm.kmem_size_max doesn't help much, > postponing that panic little farther. > > db> bt > Tracing pid 7242 tid 100772 td 0xca7a57d0 > kdb_enter(c0924e28) at kdb_enter+0x2b > panic(c093a575,1000,14000000,c17f7818,0,...) at panic+0x127 > kmem_malloc(c10680c0,1000,402,ef34e7bc,c07fb86d,...) at kmem_malloc+0x7d > page_alloc(c10613c0,1000,ef34e7af,402,0,...) at page_alloc+0x1a > slab_zalloc(c10613c0,402,c1061480,c10613c0,da68220c,...) at slab_zalloc+0xdd > uma_zone_slab(c10613c0,502) at uma_zone_slab+0xe8 > uma_zalloc_bucket(c10613c0,502) at uma_zalloc_bucket+0x15c > uma_zalloc_arg(c10613c0,0,502) at uma_zalloc_arg+0x292 > malloc(b8,c09d4ba0,502,0,0,...) at malloc+0x46 > softdep_setup_freeblocks(cc8fb18c,0,0,800,cc8fb18c,ffffffe0,ffffffff,0,0) at sof > tdep_setup_freeblocks+0x48 > ffs_truncate(c89e3990,0,0,800,c94f4300,...) at ffs_truncate+0x5cb > ffs_write(ef34ebec) at ffs_write+0x603 > VOP_WRITE_APV(c09d5960,ef34ebec) at VOP_WRITE_APV+0xce > vn_write(caba7000,ef34ecbc,c94f4300,0,ca7a57d0) at vn_write+0x1ee > dofilewrite(ca7a57d0,7,caba7000,ef34ecbc,ffffffff,...) at dofilewrite+0x77 > kern_writev(ca7a57d0,7,ef34ecbc,7e99c3c,42f6e8,...) at kern_writev+0x3b > write(ca7a57d0,ef34ed04) at write+0x45 > syscall(3b,82b003b,bfbf003b,8851,82bb000,...) at syscall+0x2bf > Xint0x80_syscall() at Xint0x80_syscall+0x1f > --- syscall (4, FreeBSD ELF32, write), eip = 0x281ae32f, esp = 0xbfbfbbdc, ebp > 0xbfbfbbf8 ---I have a high confidence that this issue should be fixed by r170991 and by minor followup in r183067. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20090805/d71a023e/attachment.pgp