Hi. I'm running i386 on i386, single P4 cpu, 1GiB RAM. SiI 3114 -> SATA [single disk] -> GELI [AES-128] -> ZFS [sha256] Straight RELENG_8 as of cvsup Oct 12 14:49:00 aka 8.0-RC1 plus. ZFS pool is at v13, ZFS fs is at v3. Hardware seems stable. The only modification to config defaults is: loader.conf.local: vfs.zfs.arc_max=100663296 After boot -v, geli, zpool import, xf86, browser, etc my mem looks like: Mem: 33M Active, 22M Inact, 105M Wired, 676K Cache, 37M Buf, 827M Free When putting load on ZFS it usually grows to about: Mem: 95M Active, 22M Inact, 302M Wired, 468K Cache, 37M Buf, 569M Free Ls -l in one of the dirs takes 10min plus and I get: PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND 11 root 171 ki31 0K 8K RUN 21:24 47.27% idle 1092 user 76 0 77328K 76116K zio->i 3:25 37.89% ls 802 root -8 - 0K 8K geli:w 1:42 8.98% g_eli[0] ad6 9 root -8 - 0K 128K arc_re 0:23 4.88% {arc_reclaim_thre} I did not watch these during rm. I have 1 parent dir holding 4 subdirs. The file count in each subdir is respectively: 256363, 254086, 256017, 178054 Two thirds of files are about 14KiB in size, not many are more than a few MiB nor less than 1KiB though a third are 1 byte. I issue rm -r <parent_dir> and after maybe 30 seconds the machine reboots. No syslog, panic or console messages. Dmesg from the prior boot is still present in ram to prove kernel didn't emit any message. memtest86 passes. There are maybe 10 seconds of complete GUI hangup before the reboot occurs. I also see it when make release'ing. Usually during what I _think_ is distributeworld or rolling up the tarballs under /R. This is a big repeatable problem. How can I debug or fix it? Can someone else create some mega sized dirs as above and replicate? Thanks.
Happened again :) So some more notes... Watched it this time, it jumped from about 29xMiB straight to 366MiB, then hung and rebooted itself. The first zpool import after reboot takes about a minute more than the usual ten seconds to import. And uses up to maybe 125MiB instead of maybe 40MiB. Could be ZFS fscking itself? Second and further manual reboots are all normal speed and mem use. The fs is fine, I can do all sorts of normal ops on it. Even rm -r <that_dir>, ^C after a few seconds, and repeat ad nauseum until the tree is removed. Only continuous rm -r reboots. I have 1 more GiB RAM I can put in. Sorry to break threads, I'm not on list.
Note: I tried setting vfs.zfs.arc_max=32M and zfs mem usage still grew past its limits and the machine rebooted. Forwarding a note I received... """"">> Your machine is starving! > How can this be, there is over 500MiB free ram at all times? I'msysctl vm.kmem_size_max I've got the following in my kernel configuration file options KVA_PAGES=512 and in /boot/loader.conf vm.kmem_size_max=1536M vm.kmem_size=1536M On two machines with 2G of RAM....both 8.0-RC1/i386 the ZFS tuning guide gives a better idea of how to play with things like that http://wiki.freebsd.org/ZFSTuningGuide """"" Some questions... Am I reading correctly that vm.kmem_size is how much ram the kernel initially allocates for itself on boot? And that vm.kmem_size_min and vm.kmem_size_max are the range that vm.kmem_size is allowed to float naturally within at runtime? Is KVA_PAGES the hard max space the kern is allowed/capable of addressing/using at runtime? Such that I could set kmem_size_max to the KVA_PAGES limit and then vm.kmem_size will grow into it as needed? With the caveat of course that with the below defaults and hardware, if I just bumped vm.kmem_size_max to 1GiB [as per KVA_PAGES limit] I'd have to add maybe another 1GiB ram so that this new vm.kmem_size_max kernel reservation wouldn't conflict with userland memory needs when vm.kmem_size grows to it? And KVA_PAGES is typically say 1/3 of installed ram? If vm.kmem_size starts out being under vm.kmem_size_max, can user apps use the unused space (vm.kmem_size_max - vm.kmem_size) until vm.kmem_size grows to vm.kmem_size_max and the kernel kills them off? Or can user apps only use (ram = user apps + [KVA_PAGES hard limit and/or vm.kmem_size_max])? What is the idea behind setting vm.kmem_size = vm.kmem_size_max? Should not just vm.kmem_size_max be set and allow vm.kmem_size [unset] to grow up to vm.kmem_size_max instead of allocating it all at boot with vm.kmem_size? Maybe someone can wikify these answers? I think I need to find more to read and then test one by one to see what changes. With untuned defaults and 1GiB ram I have: #define KVA_PAGES 256 # gives 1GiB kern address space vm.kmem_size_max: 335544320 # auto calculated by the kernel at boot? Less than KVA_PAGES? vm.kmem_size_min: 0 vm.kmem_size: 335544320 # amount in use at runtime? I'm still figuring out how to find and add all the kernel memory. Here's zfs: vfs.zfs.arc_meta_limit: 52428800 vfs.zfs.arc_meta_used: 56241732 # greater than meta_limit? vfs.zfs.arc_min: 26214400 vfs.zfs.arc_max: 209715200 vfs.zfs.vdev.cache.size: 10485760 vfs.zfs.vdev.cache.max: 16384 kstat.zfs.misc.arcstats.p: 20589785 # was 104857600 on boot kstat.zfs.misc.arcstats.c: 128292242 # was 209715200 on boot kstat.zfs.misc.arcstats.c_min: 26214400 kstat.zfs.misc.arcstats.c_max: 209715200 loader(8) vm.kmem_size Sets the size of kernel memory (bytes). This overrides the value determined when the kernel was compiled. Modifies VM_KMEM_SIZE. vm.kmem_size_min vm.kmem_size_max Sets the minimum and maximum (respectively) amount of ker- nel memory that will be automatically allocated by the ker- nel. These override the values determined when the kernel was compiled. Modifies VM_KMEM_SIZE_MIN and VM_KMEM_SIZE_MAX. sys/i386/include/pmap.h /* * Size of Kernel address space. This is the number of page table pages * (4MB each) to use for the kernel. 256 pages == 1 Gigabyte. * This **MUST** be a multiple of 4 (eg: 252, 256, 260, etc). * For PAE, the page table page unit size is 2MB. This means that 512 pages * is 1 Gigabyte. Double everything. It must be a multiple of 8 for PAE. */ #ifndef KVA_PAGES #ifdef PAE #define KVA_PAGES 512 #else #define KVA_PAGES 256 #endif #endif sys/i386/conf/NOTES # Change the size of the kernel virtual address space. Due to # constraints in loader(8) on i386, this must be a multiple of 4. # 256 = 1 GB of kernel address space. Increasing this also causes # a reduction of the address space in user processes. 512 splits # the 4GB cpu address space in half (2GB user, 2GB kernel). For PAE # kernels, the value will need to be double non-PAE. A value of 1024 # for PAE kernels is necessary to split the address space in half. # This will likely need to be increased to handle memory sizes >4GB. # PAE kernels default to a value of 512. # options KVA_PAGES=260