Xen users - Jun 2012 - Assistance in determining cause of intermittent domU crashes --stack trace(s) provided.

Hello,

I have here a Xen domU that crashes at (so far) unpredictable times, for
unknown reasons.  Following the Xen Wiki, I set it to preserve on crash and
I retrieved a stack trace via xenctx, which can be found below.
 Additionally, I connected to the console and copied whatever was there
(also below).

domU:
  openSUSE 11.2
  kernel 2.6.31.14-0.8-xen

dom0:
  openSUSE 11.4
  kernel 2.6.37.6-0.11-xen
  Motherboard:  Supermicro X7DB8
  Processors:  2x Intel Xeon E5345 @ 2.33GHz

xen:
  xen-4.0.3_01-0.2.1.x86_64 as provided by openSUSE repo.

domU configuration file:
  name = "xen-staging-app1"
  bootloader = "/usr/lib/xen/boot/domUloader.py"
  bootargs = "--entry=xvda1:/boot/vmlinuz-xen,/boot/initrd-xen"
  memory = 1536
  disk = [ ''phy:vg_xen/xen-staging-app1-root,xvda1,w'',
    ''phy:vg_xen/xen-staging-app1-swap,xvda2,w'',
    #''phy:vg/home,xvdb1,w!'',
    ''phy:vg_xen/xen-staging-app1-u-shared-data,xvdb2,w!'' ];
  root = "/dev/xvda1"
  vif = [ ''mac=00:16:3e:42:43:03, bridge=br0'' ]
  on_crash="preserve"
  extra = ""

Any help or assistance in understanding the following is greatly
appreciated.  If I can provide any other pertinent information that is not
present, please let me know and I will do my best to make it available.

Thanks,

Gavin W. Jones
Where 2 Get It, Inc.

------ xenctx stack trace ------
gjones@xen01-secondary:~> sudo /usr/lib64/xen/bin/xenctx -s
/boot/System.map-2.6.31.14-0.8-xen 51 0
rip: ffffffff800083aa hypercall_page+0x3aa
flags: 00001246 i z p
rsp: ffff88005e25a4e0
rax: 0000000000000000 rcx: ffffffff800083aa rdx: ffffffff80803060
rbx: 0000000000000000 rsi: ffff88005e25a4f8 rdi: 0000000000000002
rbp: ffff88005e25a508 r8: 0000000000000000 r9: ffffffff8000d3ea
r10: ffff88005e25a3f0 r11: 0000000000000246 r12: ffffffff80803060
r13: 00000000fffffffe r14: 0000000000000000 r15: ffffffff80747f00
 cs: e033 ss: e02b ds: 0000 es: 0000
 fs: 0000 @ 00007f6494d64790
 gs: 0000 @ ffffc90000000000/0000000000000000
Code (instr addr ffffffff800083aa)
cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b 59
c3 cc cc cc cc cc cc cc

Stack:

 ffff88005e25a128 ffffffff8073db10 ffffffff80011a8c 0000000000000003
 00000000033aa167 ffff88005e25a558 ffffffff80473c87 ffff88005e25a538
 00000000033aa167 ffffffff8073d5e6 ffffffff805a5ca2 0000000000000009
 ffff88005e25a7f8 ffff88005e258540 0000000000000001 ffff88005e25a578

Call Trace:

  [<ffffffff800083aa>] hypercall_page+0x3aa <--
  [<ffffffff80011a8c>] xen_panic_event+0x2c
  [<ffffffff80473c87>] notifier_call_chain+0x57
  [<ffffffff80473d48>] atomic_notifier_call_chain+0x28
  [<ffffffff8046ce70>] panic+0xac
  [<ffffffff80471ff1>] oops_end+0x151
  [<ffffffff80025a3b>] no_context+0x10b
  [<ffffffff80025c5d>] __bad_area_nosemaphore+0x18d
  [<ffffffff80011a39>] try_stack_unwind+0x189
  [<ffffffff80025d31>] bad_area_nosemaphore+0x21
  [<ffffffff80473b54>] do_page_fault+0x2b4
  [<ffffffff80471348>] page_fault+0x28
  [<ffffffff8003487a>] dequeue_task+0x7a
  [<ffffffff800349a8>] deactivate_task+0x38
  [<ffffffff8046dd7c>] thread_return+0x157
  [<ffffffff8046e3bd>] schedule_timeout+0x14d
  [<ffffffff8005f1f0>] process_timeout
  [<ffffffff8046d5c0>] io_schedule_timeout+0x80
  [<ffffffff8007059a>] prepare_to_wait+0x5a
  [<ffffffff800f03ca>] congestion_wait+0x7a
  [<ffffffff80070260>] autoremove_wake_function
  [<ffffffff800e645f>] do_try_to_free_pages+0x10f
  [<ffffffff800e67e7>] try_to_free_pages+0x77
  [<ffffffff800e3bd0>] isolate_pages_global
  [<ffffffff800dc183>] __alloc_pages_slowpath+0x2d3
  [<ffffffff800dc5c1>] __alloc_pages_nodemask+0x151
  [<ffffffff8010f10c>] T.583+0x4c
  [<ffffffff8010f307>] T.581+0xf7
  [<ffffffff800d69a0>] find_lock_page+0x30
  [<ffffffff8010f7f0>] cache_alloc_refill+0x260
[<ffffffff8010f99c>] kmem_cache_alloc+0x16c
-------------------------------------------------------------------------

----- Output from Console --------
[1174508.820717] Modules linked in: 8021q garp stp llc ipv6 fuse loop
dm_mod xfs exportfs xennet processor thermal_sys hwmon xenblk cdrom
[1174508.820717] Pid: 22, comm: pdflush Not tainted 2.6.31.14-0.8-xen #1
[1174508.820717] Call Trace:
[1174508.821694] [<ffffffff80011a39>] try_stack_unwind+0x189/0x1b0
[1174508.821710] [<ffffffff8000f4c6>] dump_trace+0xa6/0x1e0
[1174508.821719] [<ffffffff80011544>] show_trace_log_lvl+0x64/0x90
[1174508.821729] [<ffffffff80011593>] show_trace+0x23/0x40
[1174508.821738] [<ffffffff8046cda7>] dump_stack+0x81/0x9e
[1174508.821750] [<ffffffff8003f3d8>] __schedule_bug+0x78/0x90
[1174508.821759] [<ffffffff8046ddfc>] thread_return+0x1d7/0x3fb
[1174508.821770] [<ffffffff8046e3bd>] schedule_timeout+0x14d/0x200
[1174508.821778] [<ffffffff8046d5c0>] io_schedule_timeout+0x80/0xf0
[1174508.821790] [<ffffffff800f03ca>] congestion_wait+0x7a/0xb0
[1174508.821802] [<ffffffff800e645f>] do_try_to_free_pages+0x10f/0x380
[1174508.821810] [<ffffffff800e67e7>] try_to_free_pages+0x77/0xa0
[1174508.821822] [<ffffffff800dc183>] __alloc_pages_slowpath+0x2d3/0x5c0
[1174508.821833] [<ffffffff800dc5c1>] __alloc_pages_nodemask+0x151/0x160
[1174508.821843] [<ffffffff8010f10c>] T.583+0x4c/0x150
[1174508.821851] [<ffffffff8010f307>] T.581+0xf7/0x380
[1174508.821861] [<ffffffff8010f7f0>] cache_alloc_refill+0x260/0x2a0
[1174508.821868] [<ffffffff8010f99c>] kmem_cache_alloc+0x16c/0x180
[1174508.821904] [<ffffffffa00d0412>] kmem_zone_alloc+0xa2/0x110 [xfs]
[1174508.821982] [<ffffffffa00d5dda>] xfs_buf_get_flags+0x4a/0x1a0 [xfs]
[1174508.822057] [<ffffffffa00d5f56>] xfs_buf_read_flags+0x26/0xd0 [xfs]
[1174508.822133] [<ffffffffa00c8ae9>] xfs_trans_read_buf+0x1e9/0x320 [xfs]
[1174508.822200] [<ffffffffa0092278>] xfs_btree_read_buf_block+0x68/0xe0
[xfs]
[1174508.822243] [<ffffffffa009299e>]
xfs_btree_lookup_get_block+0x8e/0x110
[xfs]
[1174508.822286] [<ffffffffa00930af>] xfs_btree_lookup+0xdf/0x4d0 [xfs]
[1174508.822328] [<ffffffffa0079b77>] xfs_alloc_lookup_eq+0x27/0x50 [xfs]
[1174508.822356] [<ffffffffa0079f45>] xfs_alloc_fixup_trees+0x285/0x370
[xfs]
[1174508.822387] [<ffffffffa007bc30>]
xfs_alloc_ag_vextent_near+0x4e0/0x9a0
[xfs]
[1174508.822419] [<ffffffffa007c1f5>] xfs_alloc_ag_vextent+0x105/0x160
[xfs]
[1174508.822447] [<ffffffffa007ca04>] xfs_alloc_vextent+0x204/0x4b0 [xfs]
[1174508.822475] [<ffffffffa008cc8b>] xfs_bmap_btalloc+0x18b/0xae0 [xfs]
[1174508.822508] [<ffffffffa008d60f>] xfs_bmap_alloc+0x2f/0x70 [xfs]
[1174508.822541] [<ffffffffa008e272>] xfs_bmapi+0xc22/0x1320 [xfs]
[1174508.822576] [<ffffffffa00b3278>] xfs_iomap_write_allocate+0x1d8/0x3f0
[xfs]
[1174508.822629] [<ffffffffa00b40b9>] xfs_iomap+0x2c9/0x300 [xfs]
[1174508.822683] [<ffffffffa00d11e8>] xfs_map_blocks+0x38/0x60 [xfs]
[1174508.822749] [<ffffffffa00d296a>] xfs_page_state_convert+0x3fa/0x720
[xfs]
[1174508.822816] [<ffffffffa00d2e3f>] xfs_vm_writepage+0xaf/0x160 [xfs]
[1174508.822873] [<ffffffff800ddb11>] __writepage+0x21/0x60
[1174508.822881] [<ffffffff800ded65>] write_cache_pages+0x215/0x520
[1174508.822888] [<ffffffff800df0a0>] generic_writepages+0x30/0x50
[1174508.822907] [<ffffffffa00d1d3c>] xfs_vm_writepages+0x7c/0xb0 [xfs]
[1174508.822962] [<ffffffff800df0f5>] do_writepages+0x35/0x70
[1174508.822971] [<ffffffff80140e4b>] writeback_single_inode+0x10b/0x3f0
[1174508.822980] [<ffffffff80141608>] generic_sync_sb_inodes+0x1a8/0x5f0
[1174508.822987] [<ffffffff80141a95>] sync_sb_inodes+0x45/0x60
[1174508.822994] [<ffffffff80141bc4>] writeback_inodes+0x54/0x150
[1174508.823001] [<ffffffff800de4de>] background_writeout+0xbe/0x110
[1174508.823009] [<ffffffff800e00e0>] __pdflush+0x1b0/0x3b0
[1174508.823016] [<ffffffff800e0337>] pdflush+0x57/0x80
[1174508.823024] [<ffffffff8006fba6>] kthread+0xb6/0xc0
[1174508.823032] [<ffffffff8000d3ea>] child_rip+0xa/0x20
[1174508.823048] BUG: unable to handle kernel paging request at
ffffffff80794bc0
[1174508.823056] IP: [<ffffffff8003487a>] dequeue_task+0x7a/0x110
[1174508.823065] PGD 4067 PUD 1a93067 PMD 1a97067 PTE 0
[1174508.823075] Thread overran stack, or stack corrupted
[1174508.823080] Oops: 0000 [#1] SMP
[1174508.823085] last sysfs file: /sys/devices/xen/console-0/modalias
[1174508.823092] CPU 0
[1174508.823096] Modules linked in: 8021q garp stp llc ipv6 fuse loop
dm_mod xfs exportfs xennet processor thermal_sys hwmon xenblk cdrom
[1174508.823117] Pid: 22, comm: pdflush Not tainted 2.6.31.14-0.8-xen #1
[1174508.823122] RIP: e030:[<ffffffff8003487a>] [<ffffffff8003487a>]
dequeue_task+0x7a/0x110
[1174508.823130] RSP: e02b:ffff88005e25a8a0 EFLAGS: 00010006
[1174508.823135] RAX: 000000000000a380 RBX: ffff88005e258540 RCX:
ffffffff8078a000
[1174508.823141] RDX: 0000000000000001 RSI: ffff88005e258540 RDI:
ffffc9000000a380
[1174508.823146] RBP: ffff88005e25a8c0 R08: ffff88005e25a000 R09:
0000000000000000
[1174508.823152] R10: ffff88005e25a738 R11: 00000010ffffffff R12:
0000000000000000
[1174508.823158] R13: 00000001117f2972 R14: ffffffff80843c00 R15:
ffffffff80843c00
[1174508.823166] FS: 00007f6494d64790(0000) GS:ffffc90000000000(0000)
knlGS:0000000000000000
[1174508.823173] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[1174508.823177] CR2: ffffffff80794bc0 CR3: 000000004e479000 CR4:
0000000000002660
[1174508.823183] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[1174508.823189] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[1174508.823195] Process pdflush (pid: 22, threadinfo ffff88005e25a000,
task ffff88005e258540)
[1174508.823201] Stack:
[1174508.823204] ffff88005e25a8f0 00000000033aa167 0000000000000000
ffffc9000000a380
[1174508.823211] <0> ffff88005e25a8f0 ffffffff800349a8 ffffc9000000a380
00000000033aa167
[1174508.823220] <0> 00000001117f2972 ffffc9000000a380 ffff88005e25aa00
ffffffff8046dd7c
[1174508.823230] Call Trace:
[1174508.823230] [<ffffffff800349a8>] deactivate_task+0x38/0x60
[1174508.823230] [<ffffffff8046dd7c>] thread_return+0x157/0x3fb
[1174508.823230] [<ffffffff8046e3bd>] schedule_timeout+0x14d/0x200
[1174508.823230] [<ffffffff8046d5c0>] io_schedule_timeout+0x80/0xf0
[1174508.823230] [<ffffffff800f03ca>] congestion_wait+0x7a/0xb0
[1174508.823230] [<ffffffff800e645f>] do_try_to_free_pages+0x10f/0x380
[1174508.823230] [<ffffffff800e67e7>] try_to_free_pages+0x77/0xa0
[1174508.823230] [<ffffffff800dc183>] __alloc_pages_slowpath+0x2d3/0x5c0
[1174508.823230] [<ffffffff800dc5c1>] __alloc_pages_nodemask+0x151/0x160
[1174508.823230] [<ffffffff8010f10c>] T.583+0x4c/0x150
[1174508.823230] [<ffffffff8010f307>] T.581+0xf7/0x380
[1174508.823230] [<ffffffff8010f7f0>] cache_alloc_refill+0x260/0x2a0
[1174508.823230] [<ffffffff8010f99c>] kmem_cache_alloc+0x16c/0x180
[1174508.823329] [<ffffffffa00d0412>] kmem_zone_alloc+0xa2/0x110 [xfs]
[1174508.823395] [<ffffffffa00d5dda>] xfs_buf_get_flags+0x4a/0x1a0 [xfs]
[1174508.823464] [<ffffffffa00d5f56>] xfs_buf_read_flags+0x26/0xd0 [xfs]
[1174508.823534] [<ffffffffa00c8ae9>] xfs_trans_read_buf+0x1e9/0x320 [xfs]
[1174508.823596] [<ffffffffa0092278>] xfs_btree_read_buf_block+0x68/0xe0
[xfs]
[1174508.823633] [<ffffffffa009299e>]
xfs_btree_lookup_get_block+0x8e/0x110
[xfs]
[1174508.823671] [<ffffffffa00930af>] xfs_btree_lookup+0xdf/0x4d0 [xfs]
[1174508.823707] [<ffffffffa0079b77>] xfs_alloc_lookup_eq+0x27/0x50 [xfs]
[1174508.823733] [<ffffffffa0079f45>] xfs_alloc_fixup_trees+0x285/0x370
[xfs]
[1174508.823759] [<ffffffffa007bc30>]
xfs_alloc_ag_vextent_near+0x4e0/0x9a0
[xfs]
[1174508.823786] [<ffffffffa007c1f5>] xfs_alloc_ag_vextent+0x105/0x160
[xfs]
[1174508.823814] [<ffffffffa007ca04>] xfs_alloc_vextent+0x204/0x4b0 [xfs]
[1174508.823824] [<ffffffffa008cc8b>] xfs_bmap_btalloc+0x18b/0xae0 [xfs]
[1174508.823824] [<ffffffffa008d60f>] xfs_bmap_alloc+0x2f/0x70 [xfs]
[1174508.823824] [<ffffffffa008e272>] xfs_bmapi+0xc22/0x1320 [xfs]
[1174508.823824] [<ffffffffa00b3278>] xfs_iomap_write_allocate+0x1d8/0x3f0
[xfs]
[1174508.823824] [<ffffffffa00b40b9>] xfs_iomap+0x2c9/0x300 [xfs]
[1174508.823824] [<ffffffffa00d11e8>] xfs_map_blocks+0x38/0x60 [xfs]
[1174508.823824] [<ffffffffa00d296a>] xfs_page_state_convert+0x3fa/0x720
[xfs]
[1174508.823824] [<ffffffffa00d2e3f>] xfs_vm_writepage+0xaf/0x160 [xfs]
[1174508.823824] [<ffffffff800ddb11>] __writepage+0x21/0x60
[1174508.823824] [<ffffffff800ded65>] write_cache_pages+0x215/0x520
[1174508.823824] [<ffffffff800df0a0>] generic_writepages+0x30/0x50
[1174508.823824] [<ffffffffa00d1d3c>] xfs_vm_writepages+0x7c/0xb0 [xfs]
[1174508.823824] [<ffffffff800df0f5>] do_writepages+0x35/0x70
[1174508.823824] [<ffffffff80140e4b>] writeback_single_inode+0x10b/0x3f0
[1174508.823824] [<ffffffff80141608>] generic_sync_sb_inodes+0x1a8/0x5f0
[1174508.823824] [<ffffffff80141a95>] sync_sb_inodes+0x45/0x60
[1174508.823824] [<ffffffff80141bc4>] writeback_inodes+0x54/0x150
[1174508.823824] [<ffffffff800de4de>] background_writeout+0xbe/0x110
[1174508.823824] [<ffffffff800e00e0>] __pdflush+0x1b0/0x3b0
[1174508.823824] [<ffffffff800e0337>] pdflush+0x57/0x80
[1174508.823824] [<ffffffff8006fba6>] kthread+0xb6/0xc0
[1174508.823824] [<ffffffff8000d3ea>] child_rip+0xa/0x20
[1174508.823824] Code: c1 f8 03 48 01 f0 48 89 83 a0 00 00 00 4c 8b 43 08
4c 8b 8b 18 02 00 00 48 c7 c0 80 a3 00 00 41 8b 48 18 48 8b 0c cd 80 51 78
80 <48> 8b b4 08 40 08 00 00 31 c9 48 c7 83 18 02 00 00 00 00 00 00
[1174508.823824] RIP [<ffffffff8003487a>] dequeue_task+0x7a/0x110
[1174508.823824] RSP <ffff88005e25a8a0>
[1174508.823824] CR2: ffffffff80794bc0
[1174508.823824] ---[ end trace f22a09edd947e86c ]---
[1174508.823824] Kernel panic - not syncing: Fatal exception in interrupt
[1174508.823824] Pid: 22, comm: pdflush Tainted: G D 2.6.31.14-0.8-xen #1
[1174508.823824] Call Trace:
[1174508.823824] [<ffffffff80011a39>] try_stack_unwind+0x189/0x1b0
[1174508.823824] [<ffffffff8000f4c6>] dump_trace+0xa6/0x1e0
[1174508.823824] [<ffffffff80011544>] show_trace_log_lvl+0x64/0x90
[1174508.823824] [<ffffffff80011593>] show_trace+0x23/0x40
[1174508.823824] [<ffffffff8046cda7>] dump_stack+0x81/0x9e
[1174508.823824] [<ffffffff8046ce46>] panic+0x82/0x16e
[1174508.823824] [<ffffffff80471ff1>] oops_end+0x151/0x160
[1174508.823824] [<ffffffff80025a3b>] no_context+0x10b/0x1a0
[1174508.823824] [<ffffffff80025c5d>] __bad_area_nosemaphore+0x18d/0x240
[1174508.823824] [<ffffffff80025d31>] bad_area_nosemaphore+0x21/0x40
[1174508.823824] [<ffffffff80473b54>] do_page_fault+0x2b4/0x390
[1174508.823824] [<ffffffff80471348>] page_fault+0x28/0x30
[1174508.823824] [<ffffffff8003487a>] dequeue_task+0x7a/0x110
[1174508.823824] [<ffffffff800349a8>] deactivate_task+0x38/0x60
[1174508.823824] [<ffffffff8046dd7c>] thread_return+0x157/0x3fb
[1174508.823824] [<ffffffff8046e3bd>] schedule_timeout+0x14d/0x200
[1174508.823824] [<ffffffff8046d5c0>] io_schedule_timeout+0x80/0xf0
[1174508.823824] [<ffffffff800f03ca>] congestion_wait+0x7a/0xb0
[1174508.823824] [<ffffffff800e645f>] do_try_to_free_pages+0x10f/0x380
[1174508.823824] [<ffffffff800e67e7>] try_to_free_pages+0x77/0xa0
[1174508.823824] [<ffffffff800dc183>] __alloc_pages_slowpath+0x2d3/0x5c0
[1174508.823824] [<ffffffff800dc5c1>] __alloc_pages_nodemask+0x151/0x160
[1174508.823824] [<ffffffff8010f10c>] T.583+0x4c/0x150
[1174508.823824] [<ffffffff8010f307>] T.581+0xf7/0x380
[1174508.823824] [<ffffffff8010f7f0>] cache_alloc_refill+0x260/0x2a0
[1174508.823824] [<ffffffff8010f99c>] kmem_cache_alloc+0x16c/0x180
[1174508.823824] [<ffffffffa00d0412>] kmem_zone_alloc+0xa2/0x110 [xfs]
[1174508.823824] [<ffffffffa00d5dda>] xfs_buf_get_flags+0x4a/0x1a0 [xfs]
[1174508.823824] [<ffffffffa00d5f56>] xfs_buf_read_flags+0x26/0xd0 [xfs]
[1174508.823824] [<ffffffffa00c8ae9>] xfs_trans_read_buf+0x1e9/0x320 [xfs]
[1174508.823824] [<ffffffffa0092278>] xfs_btree_read_buf_block+0x68/0xe0
[xfs]
[1174508.823824] [<ffffffffa009299e>]
xfs_btree_lookup_get_block+0x8e/0x110
[xfs]
[1174508.823824] [<ffffffffa00930af>] xfs_btree_lookup+0xdf/0x4d0 [xfs]
[1174508.823824] [<ffffffffa0079b77>] xfs_alloc_lookup_eq+0x27/0x50 [xfs]
[1174508.823824] [<ffffffffa0079f45>] xfs_alloc_fixup_trees+0x285/0x370
[xfs]
[1174508.823824] [<ffffffffa007bc30>]
xfs_alloc_ag_vextent_near+0x4e0/0x9a0
[xfs]
[1174508.823824] [<ffffffffa007c1f5>] xfs_alloc_ag_vextent+0x105/0x160
[xfs]
[1174508.823824] [<ffffffffa007ca04>] xfs_alloc_vextent+0x204/0x4b0 [xfs]
[1174508.823824] [<ffffffffa008cc8b>] xfs_bmap_btalloc+0x18b/0xae0 [xfs]
[1174508.823824] [<ffffffffa008d60f>] xfs_bmap_alloc+0x2f/0x70 [xfs]
[1174508.823824] [<ffffffffa008e272>] xfs_bmapi+0xc22/0x1320 [xfs]
[1174508.823824] [<ffffffffa00b3278>] xfs_iomap_write_allocate+0x1d8/0x3f0
[xfs]
[1174508.823824] [<ffffffffa00b40b9>] xfs_iomap+0x2c9/0x300 [xfs]
[1174508.823824] [<ffffffffa00d11e8>] xfs_map_blocks+0x38/0x60 [xfs]
[1174508.823824] [<ffffffffa00d296a>] xfs_page_state_convert+0x3fa/0x720
[xfs]
[1174508.823824] [<ffffffffa00d2e3f>] xfs_vm_writepage+0xaf/0x160 [xfs]
[1174508.823824] [<ffffffff800ddb11>] __writepage+0x21/0x60
[1174508.823824] [<ffffffff800ded65>] write_cache_pages+0x215/0x520
[1174508.823824] [<ffffffff800df0a0>] generic_writepages+0x30/0x50
[1174508.823824] [<ffffffffa00d1d3c>] xfs_vm_writepages+0x7c/0xb0 [xfs]
[1174508.823824] [<ffffffff800df0f5>] do_writepages+0x35/0x70
[1174508.823824] [<ffffffff80140e4b>] writeback_single_inode+0x10b/0x3f0
[1174508.823824] [<ffffffff80141608>] generic_sync_sb_inodes+0x1a8/0x5f0
[1174508.823824] [<ffffffff80141a95>] sync_sb_inodes+0x45/0x60
[1174508.823824] [<ffffffff80141bc4>] writeback_inodes+0x54/0x150
[1174508.823824] [<ffffffff800de4de>] backgroun
---------------------------------------------------------------------------------


-- 
"There has grown up in the minds of certain groups in this country the
notion that because a man or corporation has made a profit out of the
public for a number of years, the government and the courts are charged
with the duty of guaranteeing such profit in the future, even in the face
of changing circumstances and contrary to public interest. This strange
doctrine is not supported by statute nor common law."

~Robert Heinlein


_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

Hello,

I have here a Xen domU that crashes at (so far) unpredictable times,
for unknown reasons. Following the Xen Wiki, I set it to preserve on
crash and I retrieved a stack trace via xenctx, which can be found
below. Additionally, I connected to the console and copied whatever
was there (also below).

domU:
openSUSE 11.2
kernel 2.6.31.14-0.8-xen

dom0:
openSUSE 11.4
kernel 2.6.37.6-0.11-xen
Motherboard: Supermicro X7DB8
Processors: 2x Intel Xeon E5345 @ 2.33GHz

xen:
xen-4.0.3_01-0.2.1.x86_64 as provided by openSUSE repo.

domU configuration file:
name = "xen-staging-app1"
bootloader = "/usr/lib/xen/boot/domUloader.py"
bootargs = "--entry=xvda1:/boot/vmlinuz-xen,/boot/initrd-xen"
memory = 1536
disk = [ ''phy:vg_xen/xen-staging-app1-root,xvda1,w'',
''phy:vg_xen/xen-staging-app1-swap,xvda2,w'',
#''phy:vg/home,xvdb1,w!'',
''phy:vg_xen/xen-staging-app1-u-shared-data,xvdb2,w!'' ];
root = "/dev/xvda1"
vif = [ ''mac=00:16:3e:42:43:03, bridge=br0'' ]
on_crash="preserve"
extra = ""

Any help or assistance in understanding the following is greatly
appreciated. If I can provide any other pertinent information that is
not present, please let me know and I will do my best to make it
available.

Thanks,

Gavin W. Jones
Where 2 Get It, Inc.

------ xenctx stack trace ------
gjones@xen01-secondary:~> sudo /usr/lib64/xen/bin/xenctx -s
/boot/System.map-2.6.31.14-0.8-xen 51 0
rip: ffffffff800083aa hypercall_page+0x3aa
flags: 00001246 i z p
rsp: ffff88005e25a4e0
rax: 0000000000000000 rcx: ffffffff800083aa rdx: ffffffff80803060
rbx: 0000000000000000 rsi: ffff88005e25a4f8 rdi: 0000000000000002
rbp: ffff88005e25a508 r8: 0000000000000000 r9: ffffffff8000d3ea
r10: ffff88005e25a3f0 r11: 0000000000000246 r12: ffffffff80803060
r13: 00000000fffffffe r14: 0000000000000000 r15: ffffffff80747f00
cs: e033 ss: e02b ds: 0000 es: 0000
fs: 0000 @ 00007f6494d64790
gs: 0000 @ ffffc90000000000/0000000000000000
Code (instr addr ffffffff800083aa)
cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b
59 c3 cc cc cc cc cc cc cc

Stack:

ffff88005e25a128 ffffffff8073db10 ffffffff80011a8c 0000000000000003
00000000033aa167 ffff88005e25a558 ffffffff80473c87 ffff88005e25a538
00000000033aa167 ffffffff8073d5e6 ffffffff805a5ca2 0000000000000009
ffff88005e25a7f8 ffff88005e258540 0000000000000001 ffff88005e25a578

Call Trace:

[<ffffffff800083aa>] hypercall_page+0x3aa <--
[<ffffffff80011a8c>] xen_panic_event+0x2c
[<ffffffff80473c87>] notifier_call_chain+0x57
[<ffffffff80473d48>] atomic_notifier_call_chain+0x28
[<ffffffff8046ce70>] panic+0xac
[<ffffffff80471ff1>] oops_end+0x151
[<ffffffff80025a3b>] no_context+0x10b
[<ffffffff80025c5d>] __bad_area_nosemaphore+0x18d
[<ffffffff80011a39>] try_stack_unwind+0x189
[<ffffffff80025d31>] bad_area_nosemaphore+0x21
[<ffffffff80473b54>] do_page_fault+0x2b4
[<ffffffff80471348>] page_fault+0x28
[<ffffffff8003487a>] dequeue_task+0x7a
[<ffffffff800349a8>] deactivate_task+0x38
[<ffffffff8046dd7c>] thread_return+0x157
[<ffffffff8046e3bd>] schedule_timeout+0x14d
[<ffffffff8005f1f0>] process_timeout
[<ffffffff8046d5c0>] io_schedule_timeout+0x80
[<ffffffff8007059a>] prepare_to_wait+0x5a
[<ffffffff800f03ca>] congestion_wait+0x7a
[<ffffffff80070260>] autoremove_wake_function
[<ffffffff800e645f>] do_try_to_free_pages+0x10f
[<ffffffff800e67e7>] try_to_free_pages+0x77
[<ffffffff800e3bd0>] isolate_pages_global
[<ffffffff800dc183>] __alloc_pages_slowpath+0x2d3
[<ffffffff800dc5c1>] __alloc_pages_nodemask+0x151
[<ffffffff8010f10c>] T.583+0x4c
[<ffffffff8010f307>] T.581+0xf7
[<ffffffff800d69a0>] find_lock_page+0x30
[<ffffffff8010f7f0>] cache_alloc_refill+0x260
[<ffffffff8010f99c>] kmem_cache_alloc+0x16c
-------------------------------------------------------------------------

----- Output from Console --------
[1174508.820717] Modules linked in: 8021q garp stp llc ipv6 fuse loop
dm_mod xfs exportfs xennet processor thermal_sys hwmon xenblk cdrom
[1174508.820717] Pid: 22, comm: pdflush Not tainted 2.6.31.14-0.8-xen #1
[1174508.820717] Call Trace:
[1174508.821694] [<ffffffff80011a39>] try_stack_unwind+0x189/0x1b0
[1174508.821710] [<ffffffff8000f4c6>] dump_trace+0xa6/0x1e0
[1174508.821719] [<ffffffff80011544>] show_trace_log_lvl+0x64/0x90
[1174508.821729] [<ffffffff80011593>] show_trace+0x23/0x40
[1174508.821738] [<ffffffff8046cda7>] dump_stack+0x81/0x9e
[1174508.821750] [<ffffffff8003f3d8>] __schedule_bug+0x78/0x90
[1174508.821759] [<ffffffff8046ddfc>] thread_return+0x1d7/0x3fb
[1174508.821770] [<ffffffff8046e3bd>] schedule_timeout+0x14d/0x200
[1174508.821778] [<ffffffff8046d5c0>] io_schedule_timeout+0x80/0xf0
[1174508.821790] [<ffffffff800f03ca>] congestion_wait+0x7a/0xb0
[1174508.821802] [<ffffffff800e645f>] do_try_to_free_pages+0x10f/0x380
[1174508.821810] [<ffffffff800e67e7>] try_to_free_pages+0x77/0xa0
[1174508.821822] [<ffffffff800dc183>] __alloc_pages_slowpath+0x2d3/0x5c0
[1174508.821833] [<ffffffff800dc5c1>] __alloc_pages_nodemask+0x151/0x160
[1174508.821843] [<ffffffff8010f10c>] T.583+0x4c/0x150
[1174508.821851] [<ffffffff8010f307>] T.581+0xf7/0x380
[1174508.821861] [<ffffffff8010f7f0>] cache_alloc_refill+0x260/0x2a0
[1174508.821868] [<ffffffff8010f99c>] kmem_cache_alloc+0x16c/0x180
[1174508.821904] [<ffffffffa00d0412>] kmem_zone_alloc+0xa2/0x110 [xfs]
[1174508.821982] [<ffffffffa00d5dda>] xfs_buf_get_flags+0x4a/0x1a0 [xfs]
[1174508.822057] [<ffffffffa00d5f56>] xfs_buf_read_flags+0x26/0xd0 [xfs]
[1174508.822133] [<ffffffffa00c8ae9>] xfs_trans_read_buf+0x1e9/0x320 [xfs]
[1174508.822200] [<ffffffffa0092278>] xfs_btree_read_buf_block+0x68/0xe0
[xfs]
[1174508.822243] [<ffffffffa009299e>]
xfs_btree_lookup_get_block+0x8e/0x110 [xfs]
[1174508.822286] [<ffffffffa00930af>] xfs_btree_lookup+0xdf/0x4d0 [xfs]
[1174508.822328] [<ffffffffa0079b77>] xfs_alloc_lookup_eq+0x27/0x50 [xfs]
[1174508.822356] [<ffffffffa0079f45>] xfs_alloc_fixup_trees+0x285/0x370
[xfs]
[1174508.822387] [<ffffffffa007bc30>]
xfs_alloc_ag_vextent_near+0x4e0/0x9a0 [xfs]
[1174508.822419] [<ffffffffa007c1f5>] xfs_alloc_ag_vextent+0x105/0x160
[xfs]
[1174508.822447] [<ffffffffa007ca04>] xfs_alloc_vextent+0x204/0x4b0 [xfs]
[1174508.822475] [<ffffffffa008cc8b>] xfs_bmap_btalloc+0x18b/0xae0 [xfs]
[1174508.822508] [<ffffffffa008d60f>] xfs_bmap_alloc+0x2f/0x70 [xfs]
[1174508.822541] [<ffffffffa008e272>] xfs_bmapi+0xc22/0x1320 [xfs]
[1174508.822576] [<ffffffffa00b3278>] xfs_iomap_write_allocate+0x1d8/0x3f0
[xfs]
[1174508.822629] [<ffffffffa00b40b9>] xfs_iomap+0x2c9/0x300 [xfs]
[1174508.822683] [<ffffffffa00d11e8>] xfs_map_blocks+0x38/0x60 [xfs]
[1174508.822749] [<ffffffffa00d296a>] xfs_page_state_convert+0x3fa/0x720
[xfs]
[1174508.822816] [<ffffffffa00d2e3f>] xfs_vm_writepage+0xaf/0x160 [xfs]
[1174508.822873] [<ffffffff800ddb11>] __writepage+0x21/0x60
[1174508.822881] [<ffffffff800ded65>] write_cache_pages+0x215/0x520
[1174508.822888] [<ffffffff800df0a0>] generic_writepages+0x30/0x50
[1174508.822907] [<ffffffffa00d1d3c>] xfs_vm_writepages+0x7c/0xb0 [xfs]
[1174508.822962] [<ffffffff800df0f5>] do_writepages+0x35/0x70
[1174508.822971] [<ffffffff80140e4b>] writeback_single_inode+0x10b/0x3f0
[1174508.822980] [<ffffffff80141608>] generic_sync_sb_inodes+0x1a8/0x5f0
[1174508.822987] [<ffffffff80141a95>] sync_sb_inodes+0x45/0x60
[1174508.822994] [<ffffffff80141bc4>] writeback_inodes+0x54/0x150
[1174508.823001] [<ffffffff800de4de>] background_writeout+0xbe/0x110
[1174508.823009] [<ffffffff800e00e0>] __pdflush+0x1b0/0x3b0
[1174508.823016] [<ffffffff800e0337>] pdflush+0x57/0x80
[1174508.823024] [<ffffffff8006fba6>] kthread+0xb6/0xc0
[1174508.823032] [<ffffffff8000d3ea>] child_rip+0xa/0x20
[1174508.823048] BUG: unable to handle kernel paging request at ffffffff80794bc0
[1174508.823056] IP: [<ffffffff8003487a>] dequeue_task+0x7a/0x110
[1174508.823065] PGD 4067 PUD 1a93067 PMD 1a97067 PTE 0
[1174508.823075] Thread overran stack, or stack corrupted
[1174508.823080] Oops: 0000 [#1] SMP
[1174508.823085] last sysfs file: /sys/devices/xen/console-0/modalias
[1174508.823092] CPU 0
[1174508.823096] Modules linked in: 8021q garp stp llc ipv6 fuse loop
dm_mod xfs exportfs xennet processor thermal_sys hwmon xenblk cdrom
[1174508.823117] Pid: 22, comm: pdflush Not tainted 2.6.31.14-0.8-xen #1
[1174508.823122] RIP: e030:[<ffffffff8003487a>] [<ffffffff8003487a>]
dequeue_task+0x7a/0x110
[1174508.823130] RSP: e02b:ffff88005e25a8a0 EFLAGS: 00010006
[1174508.823135] RAX: 000000000000a380 RBX: ffff88005e258540 RCX:
ffffffff8078a000
[1174508.823141] RDX: 0000000000000001 RSI: ffff88005e258540 RDI:
ffffc9000000a380
[1174508.823146] RBP: ffff88005e25a8c0 R08: ffff88005e25a000 R09:
0000000000000000
[1174508.823152] R10: ffff88005e25a738 R11: 00000010ffffffff R12:
0000000000000000
[1174508.823158] R13: 00000001117f2972 R14: ffffffff80843c00 R15:
ffffffff80843c00
[1174508.823166] FS: 00007f6494d64790(0000) GS:ffffc90000000000(0000)
knlGS:0000000000000000
[1174508.823173] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[1174508.823177] CR2: ffffffff80794bc0 CR3: 000000004e479000 CR4:
0000000000002660
[1174508.823183] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[1174508.823189] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[1174508.823195] Process pdflush (pid: 22, threadinfo
ffff88005e25a000, task ffff88005e258540)
[1174508.823201] Stack:
[1174508.823204] ffff88005e25a8f0 00000000033aa167 0000000000000000
ffffc9000000a380
[1174508.823211] <0> ffff88005e25a8f0 ffffffff800349a8
ffffc9000000a380 00000000033aa167
[1174508.823220] <0> 00000001117f2972 ffffc9000000a380
ffff88005e25aa00 ffffffff8046dd7c
[1174508.823230] Call Trace:
[1174508.823230] [<ffffffff800349a8>] deactivate_task+0x38/0x60
[1174508.823230] [<ffffffff8046dd7c>] thread_return+0x157/0x3fb
[1174508.823230] [<ffffffff8046e3bd>] schedule_timeout+0x14d/0x200
[1174508.823230] [<ffffffff8046d5c0>] io_schedule_timeout+0x80/0xf0
[1174508.823230] [<ffffffff800f03ca>] congestion_wait+0x7a/0xb0
[1174508.823230] [<ffffffff800e645f>] do_try_to_free_pages+0x10f/0x380
[1174508.823230] [<ffffffff800e67e7>] try_to_free_pages+0x77/0xa0
[1174508.823230] [<ffffffff800dc183>] __alloc_pages_slowpath+0x2d3/0x5c0
[1174508.823230] [<ffffffff800dc5c1>] __alloc_pages_nodemask+0x151/0x160
[1174508.823230] [<ffffffff8010f10c>] T.583+0x4c/0x150
[1174508.823230] [<ffffffff8010f307>] T.581+0xf7/0x380
[1174508.823230] [<ffffffff8010f7f0>] cache_alloc_refill+0x260/0x2a0
[1174508.823230] [<ffffffff8010f99c>] kmem_cache_alloc+0x16c/0x180
[1174508.823329] [<ffffffffa00d0412>] kmem_zone_alloc+0xa2/0x110 [xfs]
[1174508.823395] [<ffffffffa00d5dda>] xfs_buf_get_flags+0x4a/0x1a0 [xfs]
[1174508.823464] [<ffffffffa00d5f56>] xfs_buf_read_flags+0x26/0xd0 [xfs]
[1174508.823534] [<ffffffffa00c8ae9>] xfs_trans_read_buf+0x1e9/0x320 [xfs]
[1174508.823596] [<ffffffffa0092278>] xfs_btree_read_buf_block+0x68/0xe0
[xfs]
[1174508.823633] [<ffffffffa009299e>]
xfs_btree_lookup_get_block+0x8e/0x110 [xfs]
[1174508.823671] [<ffffffffa00930af>] xfs_btree_lookup+0xdf/0x4d0 [xfs]
[1174508.823707] [<ffffffffa0079b77>] xfs_alloc_lookup_eq+0x27/0x50 [xfs]
[1174508.823733] [<ffffffffa0079f45>] xfs_alloc_fixup_trees+0x285/0x370
[xfs]
[1174508.823759] [<ffffffffa007bc30>]
xfs_alloc_ag_vextent_near+0x4e0/0x9a0 [xfs]
[1174508.823786] [<ffffffffa007c1f5>] xfs_alloc_ag_vextent+0x105/0x160
[xfs]
[1174508.823814] [<ffffffffa007ca04>] xfs_alloc_vextent+0x204/0x4b0 [xfs]
[1174508.823824] [<ffffffffa008cc8b>] xfs_bmap_btalloc+0x18b/0xae0 [xfs]
[1174508.823824] [<ffffffffa008d60f>] xfs_bmap_alloc+0x2f/0x70 [xfs]
[1174508.823824] [<ffffffffa008e272>] xfs_bmapi+0xc22/0x1320 [xfs]
[1174508.823824] [<ffffffffa00b3278>] xfs_iomap_write_allocate+0x1d8/0x3f0
[xfs]
[1174508.823824] [<ffffffffa00b40b9>] xfs_iomap+0x2c9/0x300 [xfs]
[1174508.823824] [<ffffffffa00d11e8>] xfs_map_blocks+0x38/0x60 [xfs]
[1174508.823824] [<ffffffffa00d296a>] xfs_page_state_convert+0x3fa/0x720
[xfs]
[1174508.823824] [<ffffffffa00d2e3f>] xfs_vm_writepage+0xaf/0x160 [xfs]
[1174508.823824] [<ffffffff800ddb11>] __writepage+0x21/0x60
[1174508.823824] [<ffffffff800ded65>] write_cache_pages+0x215/0x520
[1174508.823824] [<ffffffff800df0a0>] generic_writepages+0x30/0x50
[1174508.823824] [<ffffffffa00d1d3c>] xfs_vm_writepages+0x7c/0xb0 [xfs]
[1174508.823824] [<ffffffff800df0f5>] do_writepages+0x35/0x70
[1174508.823824] [<ffffffff80140e4b>] writeback_single_inode+0x10b/0x3f0
[1174508.823824] [<ffffffff80141608>] generic_sync_sb_inodes+0x1a8/0x5f0
[1174508.823824] [<ffffffff80141a95>] sync_sb_inodes+0x45/0x60
[1174508.823824] [<ffffffff80141bc4>] writeback_inodes+0x54/0x150
[1174508.823824] [<ffffffff800de4de>] background_writeout+0xbe/0x110
[1174508.823824] [<ffffffff800e00e0>] __pdflush+0x1b0/0x3b0
[1174508.823824] [<ffffffff800e0337>] pdflush+0x57/0x80
[1174508.823824] [<ffffffff8006fba6>] kthread+0xb6/0xc0
[1174508.823824] [<ffffffff8000d3ea>] child_rip+0xa/0x20
[1174508.823824] Code: c1 f8 03 48 01 f0 48 89 83 a0 00 00 00 4c 8b 43
08 4c 8b 8b 18 02 00 00 48 c7 c0 80 a3 00 00 41 8b 48 18 48 8b 0c cd
80 51 78 80 <48> 8b b4 08 40 08 00 00 31 c9 48 c7 83 18 02 00 00 00 00
00 00
[1174508.823824] RIP [<ffffffff8003487a>] dequeue_task+0x7a/0x110
[1174508.823824] RSP <ffff88005e25a8a0>
[1174508.823824] CR2: ffffffff80794bc0
[1174508.823824] ---[ end trace f22a09edd947e86c ]---
[1174508.823824] Kernel panic - not syncing: Fatal exception in interrupt
[1174508.823824] Pid: 22, comm: pdflush Tainted: G D 2.6.31.14-0.8-xen #1
[1174508.823824] Call Trace:
[1174508.823824] [<ffffffff80011a39>] try_stack_unwind+0x189/0x1b0
[1174508.823824] [<ffffffff8000f4c6>] dump_trace+0xa6/0x1e0
[1174508.823824] [<ffffffff80011544>] show_trace_log_lvl+0x64/0x90
[1174508.823824] [<ffffffff80011593>] show_trace+0x23/0x40
[1174508.823824] [<ffffffff8046cda7>] dump_stack+0x81/0x9e
[1174508.823824] [<ffffffff8046ce46>] panic+0x82/0x16e
[1174508.823824] [<ffffffff80471ff1>] oops_end+0x151/0x160
[1174508.823824] [<ffffffff80025a3b>] no_context+0x10b/0x1a0
[1174508.823824] [<ffffffff80025c5d>] __bad_area_nosemaphore+0x18d/0x240
[1174508.823824] [<ffffffff80025d31>] bad_area_nosemaphore+0x21/0x40
[1174508.823824] [<ffffffff80473b54>] do_page_fault+0x2b4/0x390
[1174508.823824] [<ffffffff80471348>] page_fault+0x28/0x30
[1174508.823824] [<ffffffff8003487a>] dequeue_task+0x7a/0x110
[1174508.823824] [<ffffffff800349a8>] deactivate_task+0x38/0x60
[1174508.823824] [<ffffffff8046dd7c>] thread_return+0x157/0x3fb
[1174508.823824] [<ffffffff8046e3bd>] schedule_timeout+0x14d/0x200
[1174508.823824] [<ffffffff8046d5c0>] io_schedule_timeout+0x80/0xf0
[1174508.823824] [<ffffffff800f03ca>] congestion_wait+0x7a/0xb0
[1174508.823824] [<ffffffff800e645f>] do_try_to_free_pages+0x10f/0x380
[1174508.823824] [<ffffffff800e67e7>] try_to_free_pages+0x77/0xa0
[1174508.823824] [<ffffffff800dc183>] __alloc_pages_slowpath+0x2d3/0x5c0
[1174508.823824] [<ffffffff800dc5c1>] __alloc_pages_nodemask+0x151/0x160
[1174508.823824] [<ffffffff8010f10c>] T.583+0x4c/0x150
[1174508.823824] [<ffffffff8010f307>] T.581+0xf7/0x380
[1174508.823824] [<ffffffff8010f7f0>] cache_alloc_refill+0x260/0x2a0
[1174508.823824] [<ffffffff8010f99c>] kmem_cache_alloc+0x16c/0x180
[1174508.823824] [<ffffffffa00d0412>] kmem_zone_alloc+0xa2/0x110 [xfs]
[1174508.823824] [<ffffffffa00d5dda>] xfs_buf_get_flags+0x4a/0x1a0 [xfs]
[1174508.823824] [<ffffffffa00d5f56>] xfs_buf_read_flags+0x26/0xd0 [xfs]
[1174508.823824] [<ffffffffa00c8ae9>] xfs_trans_read_buf+0x1e9/0x320 [xfs]
[1174508.823824] [<ffffffffa0092278>] xfs_btree_read_buf_block+0x68/0xe0
[xfs]
[1174508.823824] [<ffffffffa009299e>]
xfs_btree_lookup_get_block+0x8e/0x110 [xfs]
[1174508.823824] [<ffffffffa00930af>] xfs_btree_lookup+0xdf/0x4d0 [xfs]
[1174508.823824] [<ffffffffa0079b77>] xfs_alloc_lookup_eq+0x27/0x50 [xfs]
[1174508.823824] [<ffffffffa0079f45>] xfs_alloc_fixup_trees+0x285/0x370
[xfs]
[1174508.823824] [<ffffffffa007bc30>]
xfs_alloc_ag_vextent_near+0x4e0/0x9a0 [xfs]
[1174508.823824] [<ffffffffa007c1f5>] xfs_alloc_ag_vextent+0x105/0x160
[xfs]
[1174508.823824] [<ffffffffa007ca04>] xfs_alloc_vextent+0x204/0x4b0 [xfs]
[1174508.823824] [<ffffffffa008cc8b>] xfs_bmap_btalloc+0x18b/0xae0 [xfs]
[1174508.823824] [<ffffffffa008d60f>] xfs_bmap_alloc+0x2f/0x70 [xfs]
[1174508.823824] [<ffffffffa008e272>] xfs_bmapi+0xc22/0x1320 [xfs]
[1174508.823824] [<ffffffffa00b3278>] xfs_iomap_write_allocate+0x1d8/0x3f0
[xfs]
[1174508.823824] [<ffffffffa00b40b9>] xfs_iomap+0x2c9/0x300 [xfs]
[1174508.823824] [<ffffffffa00d11e8>] xfs_map_blocks+0x38/0x60 [xfs]
[1174508.823824] [<ffffffffa00d296a>] xfs_page_state_convert+0x3fa/0x720
[xfs]
[1174508.823824] [<ffffffffa00d2e3f>] xfs_vm_writepage+0xaf/0x160 [xfs]
[1174508.823824] [<ffffffff800ddb11>] __writepage+0x21/0x60
[1174508.823824] [<ffffffff800ded65>] write_cache_pages+0x215/0x520
[1174508.823824] [<ffffffff800df0a0>] generic_writepages+0x30/0x50
[1174508.823824] [<ffffffffa00d1d3c>] xfs_vm_writepages+0x7c/0xb0 [xfs]
[1174508.823824] [<ffffffff800df0f5>] do_writepages+0x35/0x70
[1174508.823824] [<ffffffff80140e4b>] writeback_single_inode+0x10b/0x3f0
[1174508.823824] [<ffffffff80141608>] generic_sync_sb_inodes+0x1a8/0x5f0
[1174508.823824] [<ffffffff80141a95>] sync_sb_inodes+0x45/0x60
[1174508.823824] [<ffffffff80141bc4>] writeback_inodes+0x54/0x150
[1174508.823824] [<ffffffff800de4de>] backgroun
---------------------------------------------------------------------------------

--
"There has grown up in the minds of certain groups in this country the
notion that because a man or corporation has made a profit out of the
public for a number of years, the government and the courts are
charged with the duty of guaranteeing such profit in the future, even
in the face of changing circumstances and contrary to public interest.
This strange doctrine is not supported by statute nor common law."

~Robert Heinlein