thr3ads.net - xen discuss - 32bit dom0: kernel stack overflows with zvol block-backend device [Jan 2008]

If this information is useful, please help other people find it:
Share via:

Jürgen Keil

2008-Jan-15 17:11 UTC

32bit dom0: kernel stack overflows with zvol block-backend device

I''m observing frequent dom0 panics on

- dom0: *32-bit* snv_82,
  AMD Athlon(tm) 64 X2 Dual Core Processor 6400+,
  8GB memory

  Note: dom0 booted in 32-bit mode

- xVM version 3.0.4-1-xvm
  Latest ChangeSet: Tue Nov 27 15:29:15 2007 -0800 13229:7f0d4aba3763

- domU: PV 32-bit opensuse 10.3

  The domU is running on a 32GB virtual disk, using a compressed /
  sparse zvol block device



The opensuse 10.3 PV domU (was installed and) is configured like this:

=====================================name            = "opensuse"
memory          = "1280"

#kernel         = ''/media/SU1030.001/boot/i386/vmlinuz-xenpae''
#ramdisk                =
''/media/SU1030.001/boot/i386/initrd-xenpae''
#root           = ''xvda2''
#extra          = ''xencons=tty''

bootloader      = ''pygrub''
kernel          = ''/boot/vmlinuz-xenpae''
ramdisk         = ''/boot/initrd-xenpae''
extra           = ''xencons=tty''
root            = ''xvda2''

disk            = [ ''phy:/dev/zvol/dsk/files2/opensuse,xvda,w''
]
vif             = [ ''mac=2:c6:dd:b3:fd:7a'' ]

on_shutdown     = ''destroy''
on_reboot       = ''destroy''
on_crash        = ''destroy''
=====================================

The dom0 panic isn''t 100% reproducible, but in about one
out of five attempts to boot the opensuse domU - using
"xm create -c opensuse" - dom0 crashes when domU
starts the ext3 fsck on the root filesystem:

    ....
    Checking file systems...
    fsck 1.40.2 (12-Jul-2007)



The opensolaris dom0 dumps the following to the serial console:

=================================================================...

Xen panic[dom=0xff1ce080/vcpu=0xff27a080]: Domain 0 crashed: 
         gs:      161  fs:        0  es:     e010  ds:     e010
        edi: edf3d540 esi: c9620ff4 ebp: ff1ebf28 esp: ff1ebf0c
        ebx: ff1ebfb4 edx: ff1fe3c1 ecx: ffbf0000 eax: ff19dcab
        eip: ff14343e  cs:     e008  ss:     e010
        cr0: 8005003b  cr2: c9620ff4  cr3:  e784000  cr4:      6f0
Xen panic[dom=0xff1ce080/vcpu=0xff27a080]: Domain 0 crashed: 

ff1ebf28 xpv:sedf_dump_cpu_state+26c
ff1ebf48 xpv:__domain_crash_synchronous+54
ff1ebf68 xpv:__domain_crash+82
ff1ebf88 xpv:__domain_crash+ca
ff1ebfa8 xpv:sh_page_fault__shadow_3_guest_2+75e
c9621094 unix:page_create_io+6bc (f502be98, ce310000,)
c96210cc unix:page_create_io_wrapper+3a (ce310000, 2000, 1, )
c962110c unix:segkmem_xalloc+a3 (f50871f8, 0, 2000, )
c9621130 unix:segkmem_alloc_io_4G+22 (f50871f8, 2000, 1)
c96211cc genunix:vmem_xalloc+40f (c1412000, 2000, 100)
c9621210 genunix:vmem_alloc+126 (c1412000, 2000, 1)
c9621258 unix:kalloca+1dd (1000, 1000, 0, 0, c)
c96212ac unix:i_ddi_mem_alloc+169 (c2a13680, c96212e8,)
c962133c rootnex:rootnex_setup_copybuf+126 (e0aee000, c962141c,)
c9621398 rootnex:rootnex_bind_slowpath+2e (e0aee000, c962141c,)
c96213e8 rootnex:rootnex_dma_bindhdl+25f (c2a1be20, c2a13680,)
c9621440 genunix:ddi_dma_buf_bind_handle+f0 (e0aee000, d618ef38,)
c9621498 sata:sata_dma_buf_setup+1c9 (e00b6cb4, 40000, 0,)
c962154c sata:sata_scsi_init_pkt+197 (c41a4b50, e00b6c58,)
c9621588 scsi:scsi_init_pkt+48 (c41a4b50, 0, d618ef)
c9621600 sd:sd_setup_rw_pkt+f6 (c3cfdd00, c9621650,)
c9621654 sd:sd_initpkt_for_buf+7c (d618ef38, c9621690)
c9621694 sd:sd_start_cmds+15f (c3cfdd00, 0)
c96216b0 sd:sd_core_iostart+158 (4, c3cfdd00, d618ef)
c96216f0 sd:sd_mapblockaddr_iostart+153 (3, c3cfdd00, d618ef)
c9621710 sd:sd_xbuf_strategy+30 (d618ef38, cd5fa1e8,)
c9621740 sd:xbuf_iostart+de (c41b8ad8)
c9621758 sd:ddi_xbuf_qstrategy+4b (d618ef38, c41b8ad8)
c962177c sd:sdstrategy+dd (d618ef38)
c9621790 genunix:bdev_strategy+4d (d618ef38)
c96217a4 genunix:ldi_strategy+40 (c615d308, d618ef38)
c96217d4 zfs:vdev_disk_io_start+14a (da07d2f8)
c9621800 zfs:zio_vdev_io_start+177 (da07d2f8)
c9621814 zfs:zio_execute+66 (da07d2f8)
c9621824 zfs:zio_nowait+e (da07d2f8)
c9621854 zfs:vdev_mirror_io_start+151 (cd6918e8)
c9621880 zfs:zio_vdev_io_start+184 (cd6918e8)
c9621894 zfs:zio_execute+66 (cd6918e8)
c96218a4 zfs:zio_nowait+e (cd6918e8)
c96218e0 zfs:arc_read+6ca (0, c4147680, c62e51)
c962195c zfs:dbuf_prefetch+124 (cbdfa4b0, 1c41e2, 0)
c9621994 zfs:dmu_zfetch_fetch+48 (cbdfa4b0, 1c41e2, 0)
c96219f4 zfs:dmu_zfetch_dofetch+183 (cbdfa618, dfe78190)
c9621a40 zfs:dmu_zfetch_find+530 (cbdfa618, c9621a68,)
c9621ac4 zfs:dmu_zfetch+cc (cbdfa618, 881c4000,)
c9621afc zfs:dbuf_read+c9 (cbc4e9f8, dff5d258,)
c9621b54 zfs:dmu_buf_hold_array_by_dnode+227 (cbdfa4b0, 881c4000,)
c9621bc4 zfs:dmu_read+c9 (c91ed9f8, 1, 0, 881)
c9621c3c zfs:zvol_strategy+1e9 (df10795c)
c9621c50 genunix:bdev_strategy+4d (df10795c)
c9621c64 genunix:ldi_strategy+40 (ce697a10, df10795c)
c9621db0 xdb:xdb_biodone+65 (df10795c)
c9621dc4 genunix:biodone+71 (df10795c)
c9621e1c zfs:zvol_strategy+2aa (df10795c)
c9621e30 genunix:bdev_strategy+4d (df10795c)
c9621e44 genunix:ldi_strategy+40 (ce697a10, df10795c)
c9621f90 xdb:xdb_biodone+65 (df10795c)
c9621fa4 genunix:biodone+71 (df10795c)
c9621ffc zfs:zvol_strategy+2aa (df10795c)
c9622010 genunix:bdev_strategy+4d (df10795c)
c9622024 genunix:ldi_strategy+40 (ce697a10, df10795c)
c9622170 xdb:xdb_biodone+65 (df10795c)
c9622184 genunix:biodone+71 (df10795c)
c96221dc zfs:zvol_strategy+2aa (df10795c)
c96221f0 genunix:bdev_strategy+4d (df10795c)
c9622204 genunix:ldi_strategy+40 (ce697a10, df10795c)
c9622350 xdb:xdb_biodone+65 (df10795c)
c9622364 genunix:biodone+71 (df10795c)
c96223bc zfs:zvol_strategy+2aa (df10795c)
c96223d0 genunix:bdev_strategy+4d (df10795c)
c96223e4 genunix:ldi_strategy+40 (ce697a10, df10795c)
c9622530 xdb:xdb_biodone+65 (df10795c)
c9622544 genunix:biodone+71 (df10795c)
c962259c zfs:zvol_strategy+2aa (df10795c)
c96225b0 genunix:bdev_strategy+4d (df10795c)
c96225c4 genunix:ldi_strategy+40 (ce697a10, df10795c)
c9622710 xdb:xdb_biodone+65 (df10795c)
c9622724 genunix:biodone+71 (df10795c)
c962277c zfs:zvol_strategy+2aa (df10795c)
c9622790 genunix:bdev_strategy+4d (df10795c)
c96227a4 genunix:ldi_strategy+40 (ce697a10, df10795c)
c96228f0 xdb:xdb_biodone+65 (df10795c)
c9622904 genunix:biodone+71 (df10795c)
c962295c zfs:zvol_strategy+2aa (df10795c)
c9622970 genunix:bdev_strategy+4d (df10795c)
c9622984 genunix:ldi_strategy+40 (ce697a10, df10795c)
c9622ad0 xdb:xdb_biodone+65 (df10795c)
c9622ae4 genunix:biodone+71 (df10795c)
c9622b3c zfs:zvol_strategy+2aa (df10795c)
c9622b50 genunix:bdev_strategy+4d (df10795c)
c9622b64 genunix:ldi_strategy+40 (ce697a10, df10795c)
c9622cb0 xdb:xdb_biodone+65 (df10795c)
c9622cc4 genunix:biodone+71 (df10795c)
c9622d1c zfs:zvol_strategy+2aa (df10795c)
c9622d30 genunix:bdev_strategy+4d (df10795c)
c9622d44 genunix:ldi_strategy+40 (ce697a10, df10795c)
c9622d78 xdb:xdb_send_buf+5c (df107000, 0, 0, 0, )
c9622dc8 genunix:taskq_thread+176 (ccbe9e00, 0)
c9622dd8 unix:thread_start+8 ()


panic[cpu1]/thread=c9622de0: Fatal pagefault at 0xf4c5c218.  fault addr=0x10 
rp=0xf505b0e8
syncing file systems... 3 done
dumping to /dev/dsk/c7t0d0s1, offset 431030272, content: kernel
NOTICE: /pci@0,0/pci1043,81c0@e:
 port 0: device reset

100% done: 141123 pages dumped, compression ratio 2.64, dump succeeded
=================================================================

Unfortunately, no savable kernel crash dump seems to be detected
on the next boot, although the kernel does tell me it has dumped
something to /dev/dsk/c7t0d0s1.  Not sure why there is no saved
crash dump.  Part of the problem could be that the box has 8G of
memory, and 2 * 2GB of swap space, but even with dom0_mem=4096M
and a crash dump compression ratio > 2.0 there''s still no crash dump
saved.  



OTOH, looking at the stack frame addresses listed in the serial console
output, we see that ~ 8192 bytes of kernel stack must be in use:

  c9622dd8 - c9621094 = 0x1d44 (7492)

bytes of stack are in use.  So it seems we could have a kernel
stack overflow for that thread with the xdb:xdb_send_buf() call.

(The 32-bit x86 kernel uses 8K kernel stacks)
> _defaultstksz::print
0x2000> default_stksize::print0x2000



Hmm, the repeating pattern on the stack looks "interesting";
this may or may not be a problem:

    ....
    genunix:ldi_strategy+40 (ce697a10, df10795c)
    xdb:xdb_biodone+65 (df10795c)
    genunix:biodone+71 (df10795c)
    zfs:zvol_strategy+2aa (df10795c)
    genunix:bdev_strategy+4d (df10795c)
    ....
 
 
This message posted from opensolaris.org

Jürgen Keil

2008-Jan-16 19:18 UTC

head link

Re: 32bit dom0: kernel stack overflows with zvol block-backend device

I wrote
> I''m observing frequent dom0 panics on
> 
> - dom0: *32-bit* snv_82,
> - domU: PV 32-bit opensuse 10.3....
> ... looking at the stack frame addresses listed in
> the serial console output, we see that ~ 8192 bytes
> of kernel stack must be in use:
> 
> (The 32-bit x86 kernel uses 8K kernel stacks)

There have been no more dom0 panics today,
with the following workaround added to /etc/system:

    set _defaultstksz = 0x5000
 
 
This message posted from opensolaris.org

John Levon

2008-Jan-16 22:32 UTC

head link

Re: 32bit dom0: kernel stack overflows with zvol block-backend device

On Tue, Jan 15, 2008 at 09:11:09AM -0800, J??rgen Keil wrote:
> I''m observing frequent dom0 panics on
> 
> Hmm, the repeating pattern on the stack looks "interesting";
> this may or may not be a problem:
> 
>     ....
>     genunix:ldi_strategy+40 (ce697a10, df10795c)
>     xdb:xdb_biodone+65 (df10795c)
>     genunix:biodone+71 (df10795c)
>     zfs:zvol_strategy+2aa (df10795c)
>     genunix:bdev_strategy+4d (df10795c)
>     ....
It doesn''t look good, indeed - can you please file a bug?
We should only be recursing here a small amount, whilst we finish off an
ioreq, but it looks like BLKIF_MAX_SEGMENTS_PER_REQUEST could hurt us
pretty easily.

regards
john

Jürgen Keil

2008-Jan-21 15:54 UTC

head link

Re: 32bit dom0: kernel stack overflows with zvol block-backend device

John Levon wrote:
> On Tue, Jan 15, 2008 at 09:11:09AM -0800, Jürgen Keil wrote:
> 
> > I'm observing frequent dom0 panics on
> > 
> > Hmm, the repeating pattern on the stack looks "interesting";
> > this may or may not be a problem:
> > 
> >     ....
> >     genunix:ldi_strategy+40 (ce697a10, df10795c)
> >     xdb:xdb_biodone+65 (df10795c)
> >     genunix:biodone+71 (df10795c)
> >     zfs:zvol_strategy+2aa (df10795c)
> >     genunix:bdev_strategy+4d (df10795c)
> >     ....
> 
> It doesn't look good, indeed - can you please file a bug?
I've filed bug 6652269 for this problem:

    http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6652269
> We should only be recursing here a small amount,
> whilst we finish off an ioreq, but it looks like
> BLKIF_MAX_SEGMENTS_PER_REQUEST could hurt us
> pretty easily. 
 
This message posted from opensolaris.org
_______________________________________________
xen-discuss mailing list
xen-discuss@opensolaris.org

xen discuss - Jan 2008 - 32bit dom0: kernel stack overflows with zvol block-backend device

32bit dom0: kernel stack overflows with zvol block-backend device

Re: 32bit dom0: kernel stack overflows with zvol block-backend device

Re: 32bit dom0: kernel stack overflows with zvol block-backend device

Re: 32bit dom0: kernel stack overflows with zvol block-backend device