freebsd stable - Sep 2010 - 8.1-RELEASE: [zfs] [kmem] zfs destroy snapshot results in panic: kmem_map too small

-----------------------------------------------------------------------------
Problem          : Kernel panic "kmem_malloc(114688): kmem_map too small
Trigger          : Destroy ZFS snapshots (each bigger >80GB)
Version          : FreeBSD 8.1-RELEASE (GENERIC AMD64 but with DDB)
Workaround       : kmem / zfs arc tuning in /boot/loader.conf
System Info      : Manufacturer: Supermicro
                   Mainboard   : X8DTN
                   Cpu(s) Two  : Intel(R) Xeon(R) E5520  @ 2.27GHz
                   Memory      : 10240 MB (speed: 1066 MHz)

Storage          : mpt0 Adapter:
                   Board Name: SAS3081E
                   Board Assembly: L3-00159-02D
                   Chip Name: C1068E

24 drives are connected to mpt0: /dev/da[0 up to 23]

Each drive       :
                   512             # sectorsize
                   2000398934016   # mediasize in bytes (1.8T)
                   3907029168      # mediasize in sectors
                   243201          # Cylinders according to firmware.
                   255             # Heads according to firmware.
                   63              # Sectors according to firmware.
                   JK1130YAHUSJMT    # Disk ident.

# zpool list
NAME   SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
tank  43.5T  31.0T  12.5T    71%  ONLINE  -
-----------------------------------------------------------------------------

This server has been running since dec 2009 without downtime.
We keep 14 daily zfs snapshots. The problem of our nightly kernel panic started
when the size of each snapshot exceeded about 80 GB. A cronjob first creates
new snapshots just fine and destroys expired (older >14 days) snapshots.

During the command "zfs destroy <snapname>" the memory free seen
in top(1)
drops very quickly and soon after that this panic is generated:

panic: kmem_malloc(114688): kmem_map too small: 3307884544 total allocated
cpuid = 2
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
panic() at panic+0x182
kmem_malloc() at kmem_malloc+0x5b5
uma_large_malloc() at uma_large_malloc+0x4a
malloc() at malloc+0x14b
zio_compress_data() at zio_compress_data+0xa2
zio_write_bp_init() at zio_write_bp_init+0xc2
zio_exectute() at zio_execute+0x77
taskq_run_safe() at taskq_run_safe+0x13
taskqueue_run() at taskqueue_run+0x91
taskqueue_thread_loop() at taskqueue_thread_loop+0x3f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff81261edd30, rbp = 0 ---
panic: kmem_malloc(118784): kmem_map too small: 3307884544 total allocated
cpuid = 2

During these panics no tuning or /boot/loader.conf values where present.


Only after hitting this behaviour yesterday i created boot/loader.conf:

vm.kmem_size=18G
vfs.zfs.arc_max=6G

After which the snapshots could be destroyed without panic.


Questions:

Are users required to tune?
If so, are there reliable guidelines regarding memory and ZFS?
Are the -quickly- chosen values any good? (service window was closing)


---

Leroy

On Fri, Sep 24, 2010 at 01:24:46PM +0200, Leroy van Logchem
wrote:
>
-----------------------------------------------------------------------------
> Problem          : Kernel panic "kmem_malloc(114688): kmem_map too
small
> Trigger          : Destroy ZFS snapshots (each bigger >80GB)
> Version          : FreeBSD 8.1-RELEASE (GENERIC AMD64 but with DDB)
> ...
> panic: kmem_malloc(114688): kmem_map too small: 3307884544 total allocated
> cpuid = 2
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> panic() at panic+0x182
> kmem_malloc() at kmem_malloc+0x5b5
> uma_large_malloc() at uma_large_malloc+0x4a
> malloc() at malloc+0x14b
> zio_compress_data() at zio_compress_data+0xa2
> zio_write_bp_init() at zio_write_bp_init+0xc2
> zio_exectute() at zio_execute+0x77
> taskq_run_safe() at taskq_run_safe+0x13
> taskqueue_run() at taskqueue_run+0x91
> taskqueue_thread_loop() at taskqueue_thread_loop+0x3f
> fork_exit() at fork_exit+0x12a
> fork_trampoline() at fork_trampoline+0xe
> --- trap 0, rip = 0, rsp = 0xffffff81261edd30, rbp = 0 ---
> panic: kmem_malloc(118784): kmem_map too small: 3307884544 total allocated
> cpuid = 2
Can you please provide uname -a output?  The built date of your kernel
matters in this case.  The panic looks like uma(9) is in use, which was
enabled by default and then later reverted/removed, so this is why the
build date matters.

If "sysctl vfs.zfs.zio.use_uma" shows a value of 1, then please put
this
in your /boot/loader.conf to disable uma use:


# Disable UMA (uma(9)) for ZFS; amd64 was moved to exclusively use UMA
# on 2010/05/24.
# http://lists.freebsd.org/pipermail/freebsd-stable/2010-June/057162.html
vfs.zfs.zio.use_uma="0"


If you upgrade to RELENG_8, you won't have to set this.

As for your questions under "Questions" -- yes you have to tune, no
there aren't really "reliable guidelines" and I've been asking
for such
since ZFS came out for FreeBSD, but your values look fine.

-- 
| Jeremy Chadwick                                   jdc@parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |

on 24/09/2010 16:37 Leroy van Logchem said the
following:
> On Fri, Sep 24, 2010 at 3:04 PM, Andriy Gapon <avg@icyb.net.ua>
wrote:
>> On amd64 set your vm.kmem_size to at least the amount of available
memory that you
>> got, or even more (1.5x, 2x).
>> An easy way to do that (1x) is to set vm.kmem_size_scale="1".
>> In head this is already done automatically, MFC to stable/8 is
expected.
> 
> Thank you all for the swift answers and advice. The upcoming auto scaling
sounds
> promising. Just to be sure, does the kmem_size_scale apply when using
8.1-R?
Yes.

-- 
Andriy Gapon