While trying to speed up nanobsd builds, I mounted /usr/obj on a ramdisk and found my box crashing. Thinking it might be hardware, I tried a separate machine, but with the same results. I have 4G of ram (i386). Am I just running out of some kernel memory ? If so, is there anything I can adjust to prevent this, yet still use mfs in this way ? mdconfig -a -t malloc -s 1800M newfs /dev/md0 mount /dev/md0 /usr/obj/ time make -j4 buildworld > /var/log/build.out in the middle of the buildworld on the serial console (after adding witness etc) g_vfs_done():md0[WRITE(offset=1752924160, length=6144)]error = 28 g_vfs_done():md0[WRITE(offset=1752952832, length=4096)]error = 28 g_vfs_done():md0[WRITE(offset=1753006080, length=14336)]error = 28 g_vfs_done():md0[WRITE(offset=1753020416, length=2048)]error = 28 g_vfs_done():md0[WRITE(offset=1753202688, length=81920)]error = 28 g_vfs_done():md0[WRITE(offset=1191755776, length=131072)]error = 28 g_vfs_done():md0[WRITE(offset=1191886848, length=131072)]error = 28 g_vfs_done():md0[WRITE(offset=1192017920, length=131072)]error = 28 panic: kmem_malloc(4096): kmem_map too small: 335544320 total allocated cpuid = 1 KDB: enter: panic [thread pid 15139 tid 100160 ] Stopped at kdb_enter_why+0x3a: movl $0,kdb_why db> db> bt Tracing pid 15139 tid 100160 td 0xc7a85af0 kdb_enter_why(3231417278,3231417278,3231555983,3911968708,1,...) at kdb_enter_why+58 panic(3231555983,4096,335544320,3231555977,2000,...) at panic+310 kmem_malloc(3242659980,4096,258,3911968860,3230390816,...) at kmem_malloc+640 page_alloc(3242544416,4096,3911968847,258,3242544416,...) at page_alloc+39 slab_zalloc(3228209884,3242551688,8,3231552877,1878,...) at slab_zalloc+192 uma_zone_slab(3242551688,0,3231552877,1878,4,...) at uma_zone_slab+324 uma_zalloc_arg(3242544416,0,258,3349699312,3911969228,...) at uma_zalloc_arg+1395 ffs_vget(3333761124,922776,2,3911969228,3911969240,...) at ffs_vget+168 ufs_lookup(3911969308,3343471972,3911969744,3343471972,3911969340,...) at ufs_lookup+2555 VOP_CACHEDLOOKUP_APV(3232100544,3911969308,3911969744,3911969724,3335025920,...) at VOP_CACHEDLOOKUP_APV+165 vfs_cache_lookup(3911969440,3911969440,3911969704,3343471972,2,...) at vfs_cache_lookup+208 VOP_LOOKUP_APV(3232100544,3911969440,3349699312,3231462256,681,...) at VOP_LOOKUP_APV+165 lookup(3911969704,3231462256,198,191,3343637548,...) at lookup+1422 namei(3911969704,3911969608,96,0,3349699312,...) at namei+843 kern_stat(3349699312,672272928,0,3911969816,93,...) at kern_stat+61 stat(3349699312,3911970044,8,3231440732,3231973120,...) at stat+47 syscall(3911970104) at syscall+691 Xint0x80_syscall() at Xint0x80_syscall+32 --- syscall (188, FreeBSD ELF32, stat), eip = 134726611, esp = 3217021740, ebp = 3217021864 --- db> -------------------------------------------------------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet since 1994 www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike
On Tue, Dec 02, 2008 at 08:19:17AM -0500, Mike Tancsa wrote:> While trying to speed up nanobsd builds, I mounted /usr/obj on a > ramdisk and found my box crashing. Thinking it might be hardware, I > tried a separate machine, but with the same results. I have 4G of > ram (i386). Am I just running out of some kernel memory ? If so, is > there anything I can adjust to prevent this, yet still use mfs in this way ? > > mdconfig -a -t malloc -s 1800MYou cannot have ~ 2Gb of kernel memory allocated for md, at least not on i386.> newfs /dev/md0 > mount /dev/md0 /usr/obj/ > time make -j4 buildworld > /var/log/build.out > > > in the middle of the buildworld on the serial console (after adding > witness etc) > > g_vfs_done():md0[WRITE(offset=1752924160, length=6144)]error = 28 > g_vfs_done():md0[WRITE(offset=1752952832, length=4096)]error = 28 > g_vfs_done():md0[WRITE(offset=1753006080, length=14336)]error = 28 > g_vfs_done():md0[WRITE(offset=1753020416, length=2048)]error = 28 > g_vfs_done():md0[WRITE(offset=1753202688, length=81920)]error = 28 > g_vfs_done():md0[WRITE(offset=1191755776, length=131072)]error = 28 > g_vfs_done():md0[WRITE(offset=1191886848, length=131072)]error = 28 > g_vfs_done():md0[WRITE(offset=1192017920, length=131072)]error = 28 > panic: kmem_malloc(4096): kmem_map too small: 335544320 total allocated > cpuid = 1 > KDB: enter: panic > [thread pid 15139 tid 100160 ] > Stopped at kdb_enter_why+0x3a: movl $0,kdb_why > db> > db> bt > Tracing pid 15139 tid 100160 td 0xc7a85af0 > kdb_enter_why(3231417278,3231417278,3231555983,3911968708,1,...) at > kdb_enter_why+58 > panic(3231555983,4096,335544320,3231555977,2000,...) at panic+310 > kmem_malloc(3242659980,4096,258,3911968860,3230390816,...) at > kmem_malloc+640 > page_alloc(3242544416,4096,3911968847,258,3242544416,...) at page_alloc+39 > slab_zalloc(3228209884,3242551688,8,3231552877,1878,...) at slab_zalloc+192 > uma_zone_slab(3242551688,0,3231552877,1878,4,...) at uma_zone_slab+324 > uma_zalloc_arg(3242544416,0,258,3349699312,3911969228,...) at > uma_zalloc_arg+1395 > ffs_vget(3333761124,922776,2,3911969228,3911969240,...) at ffs_vget+168 > ufs_lookup(3911969308,3343471972,3911969744,3343471972,3911969340,...) > at ufs_lookup+2555 > VOP_CACHEDLOOKUP_APV(3232100544,3911969308,3911969744,3911969724,3335025920,...) > at VOP_CACHEDLOOKUP_APV+165 > vfs_cache_lookup(3911969440,3911969440,3911969704,3343471972,2,...) > at vfs_cache_lookup+208 > VOP_LOOKUP_APV(3232100544,3911969440,3349699312,3231462256,681,...) > at VOP_LOOKUP_APV+165 > lookup(3911969704,3231462256,198,191,3343637548,...) at lookup+1422 > namei(3911969704,3911969608,96,0,3349699312,...) at namei+843 > kern_stat(3349699312,672272928,0,3911969816,93,...) at kern_stat+61 > stat(3349699312,3911970044,8,3231440732,3231973120,...) at stat+47 > syscall(3911970104) at syscall+691 > Xint0x80_syscall() at Xint0x80_syscall+32 > --- syscall (188, FreeBSD ELF32, stat), eip = 134726611, esp = > 3217021740, ebp = 3217021864 --- > db> > > > -------------------------------------------------------------------- > Mike Tancsa, tel +1 519 651 3400 > Sentex Communications, mike@sentex.net > Providing Internet since 1994 www.sentex.net > Cambridge, Ontario Canada www.sentex.net/mike > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20081202/a363549f/attachment.pgp
At 08:38 AM 12/2/2008, Kostik Belousov wrote:> > > > mdconfig -a -t malloc -s 1800M >You cannot have ~ 2Gb of kernel memory allocated for md, at least not on >i386.Thanks, how do I find out what the limit is on a machine ? Is it vm.kvm_size ? ---Mike
On Tue, Dec 02, 2008 at 09:12:54AM -0500, Mike Tancsa wrote:> At 08:38 AM 12/2/2008, Kostik Belousov wrote: > >> > >> mdconfig -a -t malloc -s 1800M > >You cannot have ~ 2Gb of kernel memory allocated for md, at least not on > >i386. > > Thanks, how do I find out what the limit is on a machine ? Is it > vm.kvm_size ?It is much less, and highly depends on your load, since KVA is used for all kind of allocations made by kernel. I think either md(4) or mdconfig(8) have a warning about malloc backing for md. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20081202/6bb0d3a0/attachment.pgp
At 09:20 AM 12/2/2008, Kostik Belousov wrote:>On Tue, Dec 02, 2008 at 09:12:54AM -0500, Mike Tancsa wrote: > > At 08:38 AM 12/2/2008, Kostik Belousov wrote: > > >> > > >> mdconfig -a -t malloc -s 1800M > > >You cannot have ~ 2Gb of kernel memory allocated for md, at least not on > > >i386. > > > > Thanks, how do I find out what the limit is on a machine ? Is it > > vm.kvm_size ? > >It is much less, and highly depends on your load, since KVA is used for all >kind of allocations made by kernel. I think either md(4) or mdconfig(8) have >a warning about malloc backing for md.Thanks! A warning might be helpful to prevent such foot shooting :) ---Mike
On Tue, Dec 2, 2008 at 5:12 PM, Mike Tancsa <mike@sentex.net> wrote:> At 08:38 AM 12/2/2008, Kostik Belousov wrote: > >> > >> > mdconfig -a -t malloc -s 1800M >> You cannot have ~ 2Gb of kernel memory allocated for md, at least not on >> i386. >> > > Thanks, how do I find out what the limit is on a machine ? Is it > vm.kvm_size ? > > > ---Mike > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >Try tune both vm.kmem_size and vm.kmem_size_max via /boot/loader.conf. They are equal to 330M by default.
Mike Tancsa wrote:> At 09:20 AM 12/2/2008, Kostik Belousov wrote: >> On Tue, Dec 02, 2008 at 09:12:54AM -0500, Mike Tancsa wrote: >> > At 08:38 AM 12/2/2008, Kostik Belousov wrote: >> > >> >> > >> mdconfig -a -t malloc -s 1800M >> > >You cannot have ~ 2Gb of kernel memory allocated for md, at least >> not on >> > >i386. >> > >> > Thanks, how do I find out what the limit is on a machine ? Is it >> > vm.kvm_size ? >> >> It is much less, and highly depends on your load, since KVA is used >> for all >> kind of allocations made by kernel. I think either md(4) or >> mdconfig(8) have >> a warning about malloc backing for md. > > Thanks! A warning might be helpful to prevent such foot shooting :)malloc Storage for this type of memory disk is allocated with malloc(9). This limits the size to the malloc bucket limit in the kernel. If the -o reserve option is not set, creating and filling a large malloc-backed memory disk is a very easy way to panic a system. You almost never want to use malloc backing for a md, in favour of swap backing. Kris