Lars Kristiansen wrote: > > Hello again, I am the "me too"-guy with console-access. Hi Lars, I'm the one who started this thread, when I could not compile a new kernel after I upgraded to 6 (Pentium-1, 150 MHz, 32 MB Ram). How do you trigger the deadlock? The only way I know of how to trigger the deadlock, is to compile a new kernel and the 'linking kernel' stage will lock-up the PC. With a regular kernel, this takes 2.5 hours until deadlock, but with a fully equipped debug kernel it takes about 8 hours.... If you have a different (and possibly quicker) way to trigger the deadlock, I would like to try that also on my system. Please let me know. Regards, Rob. --------------------------------- Yahoo! FareChase - Search multiple travel sites in one click.
On Thu, 2005-Nov-17 00:00:03 -0800, Rob wrote:> The only way I know of how to trigger the deadlock, is to compile > a new kernel and the 'linking kernel' stage will lock-up the PC. > With a regular kernel, this takes 2.5 hours until deadlock, but with > a fully equipped debug kernel it takes about 8 hours....When the first deadlock occurs, you have a fully populated set of kernel objects (though possibly some of them are in the buffer case rather than on disk). You should be able to quickly reproduce the panic by running: # cd /usr/obj/usr/src/sys/<<KERNEL>> # make (Adjust the directory to suit your config name and MAKEOBJDIRPREFIX). Alternatively, check out the following lines in /usr/src/Makefile.inc1 # -DNO_KERNELCONFIG do not run config in ${MAKE} buildkernel # -DNO_KERNELCLEAN do not run ${MAKE} clean in ${MAKE} buildkernel # -DNO_KERNELDEPEND do not run ${MAKE} depend in ${MAKE} buildkernel -- Peter Jeremy
Peter Jeremy wrote: > On Thu, 2005-Nov-17 00:00:03 -0800, Rob wrote: > >>The only way I know of how to trigger the deadlock, is to compile >>a new kernel and the 'linking kernel' stage will lock-up the PC. >>With a regular kernel, this takes 2.5 hours until deadlock, but with >>a fully equipped debug kernel it takes about 8 hours.... > > > When the first deadlock occurs, you have a fully populated set of kernel > objects (though possibly some of them are in the buffer case rather than > on disk). You should be able to quickly reproduce the panic by running: > # cd /usr/obj/usr/src/sys/<<KERNEL>> > # make I'm doing this right now... Although I have not got into the deadlock yet, I do get these lines in the serial console: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 12333, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22990, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22928, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22625, size: 8192 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22990, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22784, size: 8192 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 23233, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 23404, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 23154, size: 4096 A typical snapshot of 'swapinfo' at this time is: Device 1K-blocks Used Avail Capacity /dev/ad0s1b 39848 39840 39848 100% /dev/md0 131072 54136 131072 41% Total 170920 93976 76944 55% (By the way, notice that the 'Avail' column of /dev/ad0s1b and /dev/md0 is wrong; for Total it's correct. Is this a general bug in 6 or only on my PC? ) Googling on the above swap_pager lines, I found this conversation: http://lists.freebsd.org/pipermail/freebsd-current/2005-September/055550.html which claims that DragonFly has fixed this problem. Regards, Rob. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
> Lars Kristiansen wrote: > > > > Hello again, I am the "me too"-guy with console-access. > > Hi Lars, > > I'm the one who started this thread, when I could not compile a new > kernel after I upgraded to 6 (Pentium-1, 150 MHz, 32 MB Ram). > > How do you trigger the deadlock? > > The only way I know of how to trigger the deadlock, is to compile > a new kernel and the 'linking kernel' stage will lock-up the PC. > With a regular kernel, this takes 2.5 hours until deadlock, but with > a fully equipped debug kernel it takes about 8 hours.... > > If you have a different (and possibly quicker) way to trigger the > deadlock, > I would like to try that also on my system. > Please let me know.Hi Rob! swapoff all other swap than md0, which is on the same disk as busy filesystems. Then I run for example: nice +19 portupgrade -frbWp libtool Did you see the debug-output I posted? Did those tell you anything? I do not understand those things. But I do have the debugscreen open after a crash if you want me too run anything. Just glad if I can be of help. -- Lars> > Regards, > Rob. > > > > > --------------------------------- > Yahoo! FareChase - Search multiple travel sites in one click.
Rob wrote:> --- Peter Jeremy <PeterJeremy at optushome.com.au>wrote:>> >> Basically, wait until your system deadlocks. BREAK >> into DDB. >> As a start, run 'show lockedvnods', 'ps'. My guess >> is that you'll see a lock that has a number ofwaiters>> which is probably the culprit. >> Use 'panic' to get a crashdump and then you can use >> kgdb to rummage around once you reboot >> >> If in doubt, post the output from the abovecommands>> here and someone will hopefully provide furtherinput.> > The output is here: > http://surfion.snu.ac.kr/~lahaye/swapfile.txtHas this kernel debug output been of any use? Or was this analysis not helping much as to why 6 has problems with swapfiles? I can dig further into the problem, if I get some more hints how to do so. Regards, Rob. __________________________________ Yahoo! Mail - PC Magazine Editors' Choice 2005 http://mail.yahoo.com
Peter Jeremy wrote:> On Thu, 2005-Nov-17 00:00:03 -0800, Rob wrote: > >>The only way I know of how to trigger the deadlock,is to compile>>a new kernel and the 'linking kernel' stage willlock-up the PC.>>With a regular kernel, this takes 2.5 hours untildeadlock, but with>>a fully equipped debug kernel it takes about 8hours....> > > When the first deadlock occurs, you have a fullypopulated set of kernel> objects (though possibly some of them are in thebuffer case rather than> on disk). You should be able to quickly reproducethe panic by running:> # cd /usr/obj/usr/src/sys/<<KERNEL>> > # makeI have cvsup'ed the sources to STABLE as of Nov. 23rd 2005. After recompiling/installing world and debug-kernel, I again get a kernel deadlock when using swapfile: http://surfion.snu.ac.kr/~lahaye/swapfile2.txt Previous deadlocks are still documented here http://surfion.snu.ac.kr/~lahaye/swapfile.txt I hope this is of use for fixing this bug in 6. If further investigation is needed, then please let me know. Regards, Rob. __________________________________ Yahoo! Mail - PC Magazine Editors' Choice 2005 http://mail.yahoo.com
Attempting to catch up with my backlog of unread email, only 12K unread messages to go ... On 24 Nov, Rob wrote:> I have cvsup'ed the sources to STABLE as of Nov. 23rd > 2005. > After recompiling/installing world and debug-kernel, > I again get a kernel deadlock when using swapfile: > http://surfion.snu.ac.kr/~lahaye/swapfile2.txt > > Previous deadlocks are still documented here > http://surfion.snu.ac.kr/~lahaye/swapfile.txt > > I hope this is of use for fixing this bug in 6. > If further investigation is needed, then please let me > know.This is a deadlock caused by memory exhaustion. The pagedaemon only has a limited number of bufs that it uses for writing dirty pages to swap to prevent it from saturating the I/O subsystem with large numbers of writes. In this case, pagedaemon is trying to free up memory by writing dirty pages, and it has used up all of its bufs and is waiting for the write requests to complete and the bufs the bufs to be returned to it. This isn't happening because md0 is stuck waiting for memory. This is a little bit suprising to me because it looks like writes to vnode backed devices are done synchronously by default. If you have a chance to test this again, a stack trace of md0 in the deadlock state would be interesting. I'd like to know where md0 is getting stuck. I wonder if pagedaemon should scan ahead and more agressively discard clean pages when it has run out of bufs to write dirty pages, especially in low memory situations. Preventing the creation of more dirty pages would be nice, but I don't know how to do that ...