Jean-Francois Malouin
2010-Nov-02 18:51 UTC
[Xen-users] Debian/squeeze: domU live migraton hangs
Hi,
In view of the problems I was having with DomU network timeout after a
live migration (I posted that problems here a while ago but never got
anything except from private emails) I finally updated my
Debian/Squeeze Dom0s last night to a new kernel, from 2.6.32-23 to
2.6.32-26.
Now live migration just hangs...Any ideas?
Xen-related Debian packages (all from Debian repositary, except drbd
which is from linbit):
~# dpkg -l \*xen\* | grep ^i
drbd8-module-2.6.32-5-xen-amd64 2:8.3.8-0+2.6.32-26 RAID 1 over tcp/ip for
Linux kernel module
libxenstore3.0 4.0.1-1 Xenstore
communications library for Xen
linux-headers-2.6.32-5-common-xen 2.6.32-26 Common header
files for Linux 2.6.32-5-xen
linux-headers-2.6.32-5-xen-amd64 2.6.32-26 Header files
for Linux 2.6.32-5-xen-amd64
linux-image-2.6.32-5-xen-amd64 2.6.32-26 Linux 2.6.32
for 64-bit PCs, Xen dom0 support
xen-hypervisor-4.0-amd64 4.0.1-1 The Xen
Hypervisor on AMD64
xen-qemu-dm-4.0 4.0.1-1 Xen Qemu Device
Model virtual machine hardware emulator
xen-tools 4.2-1 Tools to manage
Xen virtual servers
xen-utils-4.0 4.0.1-1 XEN
administrative tools
xen-utils-common 4.0.0-1 XEN
administrative tools - common files
xenstore-utils 4.0.1-1 Xenstore
utilities for Xen
xenwatch 0.5.4-2 Virtualization
utilities, mostly for Xen
~# xm info
host : dom0
release : 2.6.32-5-xen-amd64
version : #1 SMP Wed Oct 20 02:22:18 UTC 2010
machine : x86_64
nr_cpus : 8
nr_nodes : 2
cores_per_socket : 4
threads_per_core : 1
cpu_mhz : 2000
hw_caps :
bfebfbff:28100800:00000000:00001b40:009ce3bd:00000000:00000001:00000000
virt_caps : hvm hvm_directio
total_memory : 12279
free_memory : 3934
node_to_cpu : node0:0-3
node1:4-7
node_to_memory : node0:1996
node1:1938
node_to_dma32_mem : node0:1995
node1:0
max_node_id : 1
xen_major : 4
xen_minor : 0
xen_extra : .1
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x
86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
xen_commandline : dom0_mem=2048M dom0_max_vcpus=2 loglvl=all
guest_loglvl=all console=tty0
cc_compiler : gcc version 4.4.5 20100824 (prerelease) (Debian
4.4.4-11)
cc_compile_by : waldi
cc_compile_domain : debian.org
cc_compile_date : Fri Sep 3 15:38:12 UTC 2010
xend_config_format : 4
thanks
jf
--
<° >< Jean-François Malouin McConnell Brain Imaging Centre
Systems/Network Administrator Montréal Neurological Institute
3801 Rue University, Suite WB219 Montréal, Québec, H3A 2B4
Phone: 514-398-8924 Fax: 514-398-8948
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Jean-Francois Malouin
2010-Nov-02 20:40 UTC
Re: [Xen-users] Debian/squeeze: domU live migraton hangs
Replying to myself, * Jean-Francois Malouin <Jean-Francois.Malouin@bic.mni.mcgill.ca> [20101102 14:53]:> > Hi, > > In view of the problems I was having with DomU network timeout after a > live migration (I posted that problems here a while ago but never got > anything except from private emails) I finally updated my > Debian/Squeeze Dom0s last night to a new kernel, from 2.6.32-23 to > 2.6.32-26. > > Now live migration just hangs...Any ideas?Forgot to add the kernel log from the DomU (same kernel as dom0) [10765.336002] BUG: soft lockup - CPU#0 stuck for 61s! [xenwatch:16] [10765.336002] Modules linked in: xt_tcpudp iptable_filter ip_tables x_tables snd_pcm snd_timer snd soundcore snd_page_alloc evdev pcspkr ext3 jbd mbcache dm_mod raid1 md_mod xen_netfront xen_blkfront [10765.336002] CPU 0: [10765.336002] Modules linked in: xt_tcpudp iptable_filter ip_tables x_tables snd_pcm snd_timer snd soundcore snd_page_alloc evdev pcspkr ext3 jbd mbcache dm_mod raid1 md_mod xen_netfront [10765.336002] Pid: 16, comm: xenwatch Not tainted 2.6.32-5-xen-amd64 #1 [10765.336002] RIP: e030:[<ffffffff81068705>] [<ffffffff81068705>] lock_hrtimer_base+0x3a/0x3c [10765.336002] RSP: e02b:ffff8800ffaadd70 EFLAGS: 00000246 [10765.336002] RAX: ffff880002a28680 RBX: 0000000000000000 RCX: 0000000000000006 [10765.336002] RDX: ffff8800fcd70850 RSI: ffff8800ffaadda0 RDI: ffff880002a2f820 [10765.336002] RBP: ffff880002a2f820 R08: 0000000000000000 R09: 0000000000000000 [10765.336002] R10: ffff8800fcd70c50 R11: ffffffff8122b649 R12: ffff8800ffaadda0 [10765.336002] R13: 0000000000000002 R14: ffff8800fcd708a0 R15: ffff8800ffaaddf0 [10765.336002] FS: 00007fedc5e08710(0000) GS:ffff8800039bf000(0000) knlGS:0000000000000000 [10765.336002] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [10765.336002] CR2: 00007fae1857c120 CR3: 0000000002fe3000 CR4: 0000000000002660 [10765.336002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10765.336002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10765.336002] Call Trace: [10765.336002] [<ffffffff8106875b>] ? hrtimer_try_to_cancel+0x16/0x43 [10765.336002] [<ffffffff8122b649>] ? serial8250_suspend+0x0/0x48 [10765.336002] [<ffffffff81068794>] ? hrtimer_cancel+0xc/0x16 [10765.336002] [<ffffffffa0009147>] ? netfront_suspend+0x19/0x1d [xen_netfront] [10765.336002] [<ffffffff811f569b>] ? xenbus_dev_suspend+0x1f/0x3b [10765.336002] [<ffffffff81233872>] ? dpm_suspend_start+0x359/0x45b [10765.336002] [<ffffffff811f2ca0>] ? shutdown_handler+0x15f/0x25c [10765.336002] [<ffffffff8130b475>] ? mutex_lock+0xd/0x31 [10765.336002] [<ffffffff811f47ad>] ? xenwatch_thread+0x117/0x14a [10765.336002] [<ffffffff81065afe>] ? autoremove_wake_function+0x0/0x2e [10765.336002] [<ffffffff811f4696>] ? xenwatch_thread+0x0/0x14a [10765.336002] [<ffffffff81065831>] ? kthread+0x79/0x81 [10765.336002] [<ffffffff81012baa>] ? child_rip+0xa/0x20 [10765.336002] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b [10765.336002] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6 [10765.336002] [<ffffffff8102ddac>] ? pvclock_clocksource_read+0x3a/0x8b [10765.336002] [<ffffffff81012ba0>] ? child_rip+0x0/0x20 googling digged this: http://www.linux-archive.org/debian-kernel/442963-bug-600992-further-logs.html which pretty much what I''m seeing on my systems. jf> > Xen-related Debian packages (all from Debian repositary, except drbd > which is from linbit): > > ~# dpkg -l \*xen\* | grep ^i > drbd8-module-2.6.32-5-xen-amd64 2:8.3.8-0+2.6.32-26 RAID 1 over tcp/ip for Linux kernel module > libxenstore3.0 4.0.1-1 Xenstore communications library for Xen > linux-headers-2.6.32-5-common-xen 2.6.32-26 Common header files for Linux 2.6.32-5-xen > linux-headers-2.6.32-5-xen-amd64 2.6.32-26 Header files for Linux 2.6.32-5-xen-amd64 > linux-image-2.6.32-5-xen-amd64 2.6.32-26 Linux 2.6.32 for 64-bit PCs, Xen dom0 support > xen-hypervisor-4.0-amd64 4.0.1-1 The Xen Hypervisor on AMD64 > xen-qemu-dm-4.0 4.0.1-1 Xen Qemu Device Model virtual machine hardware emulator > xen-tools 4.2-1 Tools to manage Xen virtual servers > xen-utils-4.0 4.0.1-1 XEN administrative tools > xen-utils-common 4.0.0-1 XEN administrative tools - common files > xenstore-utils 4.0.1-1 Xenstore utilities for Xen > xenwatch 0.5.4-2 Virtualization utilities, mostly for Xen > > ~# xm info > host : dom0 > release : 2.6.32-5-xen-amd64 > version : #1 SMP Wed Oct 20 02:22:18 UTC 2010 > machine : x86_64 > nr_cpus : 8 > nr_nodes : 2 > cores_per_socket : 4 > threads_per_core : 1 > cpu_mhz : 2000 > hw_caps : bfebfbff:28100800:00000000:00001b40:009ce3bd:00000000:00000001:00000000 > virt_caps : hvm hvm_directio > total_memory : 12279 > free_memory : 3934 > node_to_cpu : node0:0-3 > node1:4-7 > node_to_memory : node0:1996 > node1:1938 > node_to_dma32_mem : node0:1995 > node1:0 > max_node_id : 1 > xen_major : 4 > xen_minor : 0 > xen_extra : .1 > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x 86_32p hvm-3.0-x86_64 > xen_scheduler : credit > xen_pagesize : 4096 > platform_params : virt_start=0xffff800000000000 > xen_changeset : unavailable > xen_commandline : dom0_mem=2048M dom0_max_vcpus=2 loglvl=all guest_loglvl=all console=tty0 > cc_compiler : gcc version 4.4.5 20100824 (prerelease) (Debian 4.4.4-11) > cc_compile_by : waldi > cc_compile_domain : debian.org > cc_compile_date : Fri Sep 3 15:38:12 UTC 2010 > xend_config_format : 4 > > thanks > jf > -- > <° >< Jean-François Malouin McConnell Brain Imaging Centre > Systems/Network Administrator Montréal Neurological Institute > 3801 Rue University, Suite WB219 Montréal, Québec, H3A 2B4 > Phone: 514-398-8924 Fax: 514-398-8948 > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jean-Francois Malouin
2010-Nov-03 21:41 UTC
Re: [Xen-users] Debian/squeeze: domU live migraton hangs
* Jean-Francois Malouin <Jean-Francois.Malouin@bic.mni.mcgill.ca> [20101102 16:41]:> Replying to myself,Continuing my soliloque, Updated Dom0 kernel to debian/squeeze to 2.6.32-27 (from -26) Looks like this resolves the save/migration issue. Still experiencing some delays for the domUs network to come up though... jf> > * Jean-Francois Malouin <Jean-Francois.Malouin@bic.mni.mcgill.ca> [20101102 14:53]: > > > > Hi, > > > > In view of the problems I was having with DomU network timeout after a > > live migration (I posted that problems here a while ago but never got > > anything except from private emails) I finally updated my > > Debian/Squeeze Dom0s last night to a new kernel, from 2.6.32-23 to > > 2.6.32-26. > > > > Now live migration just hangs...Any ideas? > > Forgot to add the kernel log from the DomU (same kernel as dom0) > > [10765.336002] BUG: soft lockup - CPU#0 stuck for 61s! [xenwatch:16] > [10765.336002] Modules linked in: xt_tcpudp iptable_filter ip_tables x_tables snd_pcm snd_timer snd soundcore snd_page_alloc evdev pcspkr ext3 jbd mbcache dm_mod raid1 md_mod xen_netfront xen_blkfront > [10765.336002] CPU 0: > [10765.336002] Modules linked in: xt_tcpudp iptable_filter ip_tables x_tables snd_pcm snd_timer snd soundcore snd_page_alloc evdev pcspkr ext3 jbd mbcache dm_mod raid1 md_mod xen_netfront > [10765.336002] Pid: 16, comm: xenwatch Not tainted 2.6.32-5-xen-amd64 #1 > [10765.336002] RIP: e030:[<ffffffff81068705>] [<ffffffff81068705>] lock_hrtimer_base+0x3a/0x3c > [10765.336002] RSP: e02b:ffff8800ffaadd70 EFLAGS: 00000246 > [10765.336002] RAX: ffff880002a28680 RBX: 0000000000000000 RCX: 0000000000000006 > [10765.336002] RDX: ffff8800fcd70850 RSI: ffff8800ffaadda0 RDI: ffff880002a2f820 > [10765.336002] RBP: ffff880002a2f820 R08: 0000000000000000 R09: 0000000000000000 > [10765.336002] R10: ffff8800fcd70c50 R11: ffffffff8122b649 R12: ffff8800ffaadda0 > [10765.336002] R13: 0000000000000002 R14: ffff8800fcd708a0 R15: ffff8800ffaaddf0 > [10765.336002] FS: 00007fedc5e08710(0000) GS:ffff8800039bf000(0000) knlGS:0000000000000000 > [10765.336002] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > [10765.336002] CR2: 00007fae1857c120 CR3: 0000000002fe3000 CR4: 0000000000002660 > [10765.336002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [10765.336002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [10765.336002] Call Trace: > [10765.336002] [<ffffffff8106875b>] ? hrtimer_try_to_cancel+0x16/0x43 > [10765.336002] [<ffffffff8122b649>] ? serial8250_suspend+0x0/0x48 > [10765.336002] [<ffffffff81068794>] ? hrtimer_cancel+0xc/0x16 > [10765.336002] [<ffffffffa0009147>] ? netfront_suspend+0x19/0x1d [xen_netfront] > [10765.336002] [<ffffffff811f569b>] ? xenbus_dev_suspend+0x1f/0x3b > [10765.336002] [<ffffffff81233872>] ? dpm_suspend_start+0x359/0x45b > [10765.336002] [<ffffffff811f2ca0>] ? shutdown_handler+0x15f/0x25c > [10765.336002] [<ffffffff8130b475>] ? mutex_lock+0xd/0x31 > [10765.336002] [<ffffffff811f47ad>] ? xenwatch_thread+0x117/0x14a > [10765.336002] [<ffffffff81065afe>] ? autoremove_wake_function+0x0/0x2e > [10765.336002] [<ffffffff811f4696>] ? xenwatch_thread+0x0/0x14a > [10765.336002] [<ffffffff81065831>] ? kthread+0x79/0x81 > [10765.336002] [<ffffffff81012baa>] ? child_rip+0xa/0x20 > [10765.336002] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b > [10765.336002] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6 > [10765.336002] [<ffffffff8102ddac>] ? pvclock_clocksource_read+0x3a/0x8b > [10765.336002] [<ffffffff81012ba0>] ? child_rip+0x0/0x20 > > googling digged this: > > http://www.linux-archive.org/debian-kernel/442963-bug-600992-further-logs.html > > which pretty much what I''m seeing on my systems. > jf > > > > > > Xen-related Debian packages (all from Debian repositary, except drbd > > which is from linbit): > > > > ~# dpkg -l \*xen\* | grep ^i > > drbd8-module-2.6.32-5-xen-amd64 2:8.3.8-0+2.6.32-26 RAID 1 over tcp/ip for Linux kernel module > > libxenstore3.0 4.0.1-1 Xenstore communications library for Xen > > linux-headers-2.6.32-5-common-xen 2.6.32-26 Common header files for Linux 2.6.32-5-xen > > linux-headers-2.6.32-5-xen-amd64 2.6.32-26 Header files for Linux 2.6.32-5-xen-amd64 > > linux-image-2.6.32-5-xen-amd64 2.6.32-26 Linux 2.6.32 for 64-bit PCs, Xen dom0 support > > xen-hypervisor-4.0-amd64 4.0.1-1 The Xen Hypervisor on AMD64 > > xen-qemu-dm-4.0 4.0.1-1 Xen Qemu Device Model virtual machine hardware emulator > > xen-tools 4.2-1 Tools to manage Xen virtual servers > > xen-utils-4.0 4.0.1-1 XEN administrative tools > > xen-utils-common 4.0.0-1 XEN administrative tools - common files > > xenstore-utils 4.0.1-1 Xenstore utilities for Xen > > xenwatch 0.5.4-2 Virtualization utilities, mostly for Xen > > > > ~# xm info > > host : dom0 > > release : 2.6.32-5-xen-amd64 > > version : #1 SMP Wed Oct 20 02:22:18 UTC 2010 > > machine : x86_64 > > nr_cpus : 8 > > nr_nodes : 2 > > cores_per_socket : 4 > > threads_per_core : 1 > > cpu_mhz : 2000 > > hw_caps : bfebfbff:28100800:00000000:00001b40:009ce3bd:00000000:00000001:00000000 > > virt_caps : hvm hvm_directio > > total_memory : 12279 > > free_memory : 3934 > > node_to_cpu : node0:0-3 > > node1:4-7 > > node_to_memory : node0:1996 > > node1:1938 > > node_to_dma32_mem : node0:1995 > > node1:0 > > max_node_id : 1 > > xen_major : 4 > > xen_minor : 0 > > xen_extra : .1 > > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x 86_32p hvm-3.0-x86_64 > > xen_scheduler : credit > > xen_pagesize : 4096 > > platform_params : virt_start=0xffff800000000000 > > xen_changeset : unavailable > > xen_commandline : dom0_mem=2048M dom0_max_vcpus=2 loglvl=all guest_loglvl=all console=tty0 > > cc_compiler : gcc version 4.4.5 20100824 (prerelease) (Debian 4.4.4-11) > > cc_compile_by : waldi > > cc_compile_domain : debian.org > > cc_compile_date : Fri Sep 3 15:38:12 UTC 2010 > > xend_config_format : 4 > > > > thanks > > jf > > -- > > <° >< Jean-François Malouin McConnell Brain Imaging Centre > > Systems/Network Administrator Montréal Neurological Institute > > 3801 Rue University, Suite WB219 Montréal, Québec, H3A 2B4 > > Phone: 514-398-8924 Fax: 514-398-8948 > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users-- <° >< Jean-François Malouin McConnell Brain Imaging Centre Systems/Network Administrator Montréal Neurological Institute 3801 Rue University, Suite WB219 Montréal, Québec, H3A 2B4 Phone: 514-398-8924 Fax: 514-398-8948 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Olivier Hanesse
2010-Nov-11 11:24 UTC
Re: [Xen-users] Debian/squeeze: domU live migraton hangs
Hello, Did you still got some delays for the domUs network to come up though ? I got some, around 90s ... After that delay, everything is working fine. any ideas ? Regards 2010/11/3 Jean-Francois Malouin <Jean-Francois.Malouin@bic.mni.mcgill.ca>> * Jean-Francois Malouin <Jean-Francois.Malouin@bic.mni.mcgill.ca> > [20101102 16:41]: > > Replying to myself, > > Continuing my soliloque, > > Updated Dom0 kernel to debian/squeeze to 2.6.32-27 (from -26) > Looks like this resolves the save/migration issue. > Still experiencing some delays for the domUs network to come up though... > > jf > > > > > * Jean-Francois Malouin <Jean-Francois.Malouin@bic.mni.mcgill.ca> > [20101102 14:53]: > > > > > > Hi, > > > > > > In view of the problems I was having with DomU network timeout after a > > > live migration (I posted that problems here a while ago but never got > > > anything except from private emails) I finally updated my > > > Debian/Squeeze Dom0s last night to a new kernel, from 2.6.32-23 to > > > 2.6.32-26. > > > > > > Now live migration just hangs...Any ideas? > > > > Forgot to add the kernel log from the DomU (same kernel as dom0) > > > > [10765.336002] BUG: soft lockup - CPU#0 stuck for 61s! [xenwatch:16] > > [10765.336002] Modules linked in: xt_tcpudp iptable_filter ip_tables > x_tables snd_pcm snd_timer snd soundcore snd_page_alloc evdev pcspkr ext3 > jbd mbcache dm_mod raid1 md_mod xen_netfront xen_blkfront > > [10765.336002] CPU 0: > > [10765.336002] Modules linked in: xt_tcpudp iptable_filter ip_tables > x_tables snd_pcm snd_timer snd soundcore snd_page_alloc evdev pcspkr ext3 > jbd mbcache dm_mod raid1 md_mod xen_netfront > > [10765.336002] Pid: 16, comm: xenwatch Not tainted 2.6.32-5-xen-amd64 #1 > > [10765.336002] RIP: e030:[<ffffffff81068705>] [<ffffffff81068705>] > lock_hrtimer_base+0x3a/0x3c > > [10765.336002] RSP: e02b:ffff8800ffaadd70 EFLAGS: 00000246 > > [10765.336002] RAX: ffff880002a28680 RBX: 0000000000000000 RCX: > 0000000000000006 > > [10765.336002] RDX: ffff8800fcd70850 RSI: ffff8800ffaadda0 RDI: > ffff880002a2f820 > > [10765.336002] RBP: ffff880002a2f820 R08: 0000000000000000 R09: > 0000000000000000 > > [10765.336002] R10: ffff8800fcd70c50 R11: ffffffff8122b649 R12: > ffff8800ffaadda0 > > [10765.336002] R13: 0000000000000002 R14: ffff8800fcd708a0 R15: > ffff8800ffaaddf0 > > [10765.336002] FS: 00007fedc5e08710(0000) GS:ffff8800039bf000(0000) > knlGS:0000000000000000 > > [10765.336002] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > > [10765.336002] CR2: 00007fae1857c120 CR3: 0000000002fe3000 CR4: > 0000000000002660 > > [10765.336002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > > [10765.336002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > > [10765.336002] Call Trace: > > [10765.336002] [<ffffffff8106875b>] ? hrtimer_try_to_cancel+0x16/0x43 > > [10765.336002] [<ffffffff8122b649>] ? serial8250_suspend+0x0/0x48 > > [10765.336002] [<ffffffff81068794>] ? hrtimer_cancel+0xc/0x16 > > [10765.336002] [<ffffffffa0009147>] ? netfront_suspend+0x19/0x1d > [xen_netfront] > > [10765.336002] [<ffffffff811f569b>] ? xenbus_dev_suspend+0x1f/0x3b > > [10765.336002] [<ffffffff81233872>] ? dpm_suspend_start+0x359/0x45b > > [10765.336002] [<ffffffff811f2ca0>] ? shutdown_handler+0x15f/0x25c > > [10765.336002] [<ffffffff8130b475>] ? mutex_lock+0xd/0x31 > > [10765.336002] [<ffffffff811f47ad>] ? xenwatch_thread+0x117/0x14a > > [10765.336002] [<ffffffff81065afe>] ? autoremove_wake_function+0x0/0x2e > > [10765.336002] [<ffffffff811f4696>] ? xenwatch_thread+0x0/0x14a > > [10765.336002] [<ffffffff81065831>] ? kthread+0x79/0x81 > > [10765.336002] [<ffffffff81012baa>] ? child_rip+0xa/0x20 > > [10765.336002] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b > > [10765.336002] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6 > > [10765.336002] [<ffffffff8102ddac>] ? > pvclock_clocksource_read+0x3a/0x8b > > [10765.336002] [<ffffffff81012ba0>] ? child_rip+0x0/0x20 > > > > googling digged this: > > > > > http://www.linux-archive.org/debian-kernel/442963-bug-600992-further-logs.html > > > > which pretty much what I''m seeing on my systems. > > jf > > > > > > > > > > Xen-related Debian packages (all from Debian repositary, except drbd > > > which is from linbit): > > > > > > ~# dpkg -l \*xen\* | grep ^i > > > drbd8-module-2.6.32-5-xen-amd64 2:8.3.8-0+2.6.32-26 RAID 1 over > tcp/ip for Linux kernel module > > > libxenstore3.0 4.0.1-1 > Xenstore communications library for Xen > > > linux-headers-2.6.32-5-common-xen 2.6.32-26 > Common header files for Linux 2.6.32-5-xen > > > linux-headers-2.6.32-5-xen-amd64 2.6.32-26 > Header files for Linux 2.6.32-5-xen-amd64 > > > linux-image-2.6.32-5-xen-amd64 2.6.32-26 Linux > 2.6.32 for 64-bit PCs, Xen dom0 support > > > xen-hypervisor-4.0-amd64 4.0.1-1 The Xen > Hypervisor on AMD64 > > > xen-qemu-dm-4.0 4.0.1-1 Xen > Qemu Device Model virtual machine hardware emulator > > > xen-tools 4.2-1 Tools to > manage Xen virtual servers > > > xen-utils-4.0 4.0.1-1 XEN > administrative tools > > > xen-utils-common 4.0.0-1 XEN > administrative tools - common files > > > xenstore-utils 4.0.1-1 > Xenstore utilities for Xen > > > xenwatch 0.5.4-2 > Virtualization utilities, mostly for Xen > > > > > > ~# xm info > > > host : dom0 > > > release : 2.6.32-5-xen-amd64 > > > version : #1 SMP Wed Oct 20 02:22:18 UTC 2010 > > > machine : x86_64 > > > nr_cpus : 8 > > > nr_nodes : 2 > > > cores_per_socket : 4 > > > threads_per_core : 1 > > > cpu_mhz : 2000 > > > hw_caps : > bfebfbff:28100800:00000000:00001b40:009ce3bd:00000000:00000001:00000000 > > > virt_caps : hvm hvm_directio > > > total_memory : 12279 > > > free_memory : 3934 > > > node_to_cpu : node0:0-3 > > > node1:4-7 > > > node_to_memory : node0:1996 > > > node1:1938 > > > node_to_dma32_mem : node0:1995 > > > node1:0 > > > max_node_id : 1 > > > xen_major : 4 > > > xen_minor : 0 > > > xen_extra : .1 > > > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 > hvm-3.0-x 86_32p hvm-3.0-x86_64 > > > xen_scheduler : credit > > > xen_pagesize : 4096 > > > platform_params : virt_start=0xffff800000000000 > > > xen_changeset : unavailable > > > xen_commandline : dom0_mem=2048M dom0_max_vcpus=2 loglvl=all > guest_loglvl=all console=tty0 > > > cc_compiler : gcc version 4.4.5 20100824 (prerelease) > (Debian 4.4.4-11) > > > cc_compile_by : waldi > > > cc_compile_domain : debian.org > > > cc_compile_date : Fri Sep 3 15:38:12 UTC 2010 > > > xend_config_format : 4 > > > > > > thanks > > > jf > > > -- > > > <° >< Jean-François Malouin McConnell Brain Imaging Centre > > > Systems/Network Administrator Montréal Neurological Institute > > > 3801 Rue University, Suite WB219 Montréal, Québec, H3A 2B4 > > > Phone: 514-398-8924 Fax: 514-398-8948 > > > > > > _______________________________________________ > > > Xen-users mailing list > > > Xen-users@lists.xensource.com > > > http://lists.xensource.com/xen-users > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > > -- > <° >< Jean-François Malouin McConnell Brain Imaging Centre > Systems/Network Administrator Montréal Neurological Institute > 3801 Rue University, Suite WB219 Montréal, Québec, H3A 2B4 > Phone: 514-398-8924 Fax: 514-398-8948 > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jean-Francois Malouin
2010-Nov-12 16:50 UTC
Re: [Xen-users] Debian/squeeze: domU live migraton hangs
Hi, * Olivier Hanesse <olivier.hanesse@gmail.com> [20101111 06:25]:> Hello, > > Did you still got some delays for the domUs network to come up though ? > I got some, around 90s ... After that delay, everything is working fine.The migration itself takes less than 60s im my tests so far and I can connect to the console immediately (or over ssh). I noticed however that sometimes after the migration is done the domUs network just hang, ie from the xend.log I see the migration is completed but the DomU network hangs or doesn''t start, taking sometimes very very long times to emerges --hours sometimes-- if it does. I know for a fact that it''s up as the uptime on the domU when it finally resurfaces shows it. As you have remarked before , it seems to sort of depend on the uptime of the dom0s, but I haven''t any conclusive evidence of that yet. I have done tcpdumps of the network traffic between the dom0s while migrating but nothing jumps out of the ordinary: I thought that a gratuitous arp request was sent by the domU? Is this still the case? I haven''t seen any... Right now both dom0s have 5+ days of uptime and things work flawlessly. Tomorrow, I''ll reboot one of them and see if I can reproduce my earlier problems. Thank you for your time! jf> > any ideas ? > > Regards > > 2010/11/3 Jean-Francois Malouin <Jean-Francois.Malouin@bic.mni.mcgill.ca> > > > * Jean-Francois Malouin <Jean-Francois.Malouin@bic.mni.mcgill.ca> > > [20101102 16:41]: > > > Replying to myself, > > > > Continuing my soliloque, > > > > Updated Dom0 kernel to debian/squeeze to 2.6.32-27 (from -26) > > Looks like this resolves the save/migration issue. > > Still experiencing some delays for the domUs network to come up though... > > > > jf > > > > > > > > * Jean-Francois Malouin <Jean-Francois.Malouin@bic.mni.mcgill.ca> > > [20101102 14:53]: > > > > > > > > Hi, > > > > > > > > In view of the problems I was having with DomU network timeout after a > > > > live migration (I posted that problems here a while ago but never got > > > > anything except from private emails) I finally updated my > > > > Debian/Squeeze Dom0s last night to a new kernel, from 2.6.32-23 to > > > > 2.6.32-26. > > > > > > > > Now live migration just hangs...Any ideas? > > > > > > Forgot to add the kernel log from the DomU (same kernel as dom0) > > > > > > [10765.336002] BUG: soft lockup - CPU#0 stuck for 61s! [xenwatch:16] > > > [10765.336002] Modules linked in: xt_tcpudp iptable_filter ip_tables > > x_tables snd_pcm snd_timer snd soundcore snd_page_alloc evdev pcspkr ext3 > > jbd mbcache dm_mod raid1 md_mod xen_netfront xen_blkfront > > > [10765.336002] CPU 0: > > > [10765.336002] Modules linked in: xt_tcpudp iptable_filter ip_tables > > x_tables snd_pcm snd_timer snd soundcore snd_page_alloc evdev pcspkr ext3 > > jbd mbcache dm_mod raid1 md_mod xen_netfront > > > [10765.336002] Pid: 16, comm: xenwatch Not tainted 2.6.32-5-xen-amd64 #1 > > > [10765.336002] RIP: e030:[<ffffffff81068705>] [<ffffffff81068705>] > > lock_hrtimer_base+0x3a/0x3c > > > [10765.336002] RSP: e02b:ffff8800ffaadd70 EFLAGS: 00000246 > > > [10765.336002] RAX: ffff880002a28680 RBX: 0000000000000000 RCX: > > 0000000000000006 > > > [10765.336002] RDX: ffff8800fcd70850 RSI: ffff8800ffaadda0 RDI: > > ffff880002a2f820 > > > [10765.336002] RBP: ffff880002a2f820 R08: 0000000000000000 R09: > > 0000000000000000 > > > [10765.336002] R10: ffff8800fcd70c50 R11: ffffffff8122b649 R12: > > ffff8800ffaadda0 > > > [10765.336002] R13: 0000000000000002 R14: ffff8800fcd708a0 R15: > > ffff8800ffaaddf0 > > > [10765.336002] FS: 00007fedc5e08710(0000) GS:ffff8800039bf000(0000) > > knlGS:0000000000000000 > > > [10765.336002] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > > > [10765.336002] CR2: 00007fae1857c120 CR3: 0000000002fe3000 CR4: > > 0000000000002660 > > > [10765.336002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > 0000000000000000 > > > [10765.336002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > > 0000000000000400 > > > [10765.336002] Call Trace: > > > [10765.336002] [<ffffffff8106875b>] ? hrtimer_try_to_cancel+0x16/0x43 > > > [10765.336002] [<ffffffff8122b649>] ? serial8250_suspend+0x0/0x48 > > > [10765.336002] [<ffffffff81068794>] ? hrtimer_cancel+0xc/0x16 > > > [10765.336002] [<ffffffffa0009147>] ? netfront_suspend+0x19/0x1d > > [xen_netfront] > > > [10765.336002] [<ffffffff811f569b>] ? xenbus_dev_suspend+0x1f/0x3b > > > [10765.336002] [<ffffffff81233872>] ? dpm_suspend_start+0x359/0x45b > > > [10765.336002] [<ffffffff811f2ca0>] ? shutdown_handler+0x15f/0x25c > > > [10765.336002] [<ffffffff8130b475>] ? mutex_lock+0xd/0x31 > > > [10765.336002] [<ffffffff811f47ad>] ? xenwatch_thread+0x117/0x14a > > > [10765.336002] [<ffffffff81065afe>] ? autoremove_wake_function+0x0/0x2e > > > [10765.336002] [<ffffffff811f4696>] ? xenwatch_thread+0x0/0x14a > > > [10765.336002] [<ffffffff81065831>] ? kthread+0x79/0x81 > > > [10765.336002] [<ffffffff81012baa>] ? child_rip+0xa/0x20 > > > [10765.336002] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b > > > [10765.336002] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6 > > > [10765.336002] [<ffffffff8102ddac>] ? > > pvclock_clocksource_read+0x3a/0x8b > > > [10765.336002] [<ffffffff81012ba0>] ? child_rip+0x0/0x20 > > > > > > googling digged this: > > > > > > > > http://www.linux-archive.org/debian-kernel/442963-bug-600992-further-logs.html > > > > > > which pretty much what I''m seeing on my systems. > > > jf > > > > > > > > > > > > > > Xen-related Debian packages (all from Debian repositary, except drbd > > > > which is from linbit): > > > > > > > > ~# dpkg -l \*xen\* | grep ^i > > > > drbd8-module-2.6.32-5-xen-amd64 2:8.3.8-0+2.6.32-26 RAID 1 over > > tcp/ip for Linux kernel module > > > > libxenstore3.0 4.0.1-1 > > Xenstore communications library for Xen > > > > linux-headers-2.6.32-5-common-xen 2.6.32-26 > > Common header files for Linux 2.6.32-5-xen > > > > linux-headers-2.6.32-5-xen-amd64 2.6.32-26 > > Header files for Linux 2.6.32-5-xen-amd64 > > > > linux-image-2.6.32-5-xen-amd64 2.6.32-26 Linux > > 2.6.32 for 64-bit PCs, Xen dom0 support > > > > xen-hypervisor-4.0-amd64 4.0.1-1 The Xen > > Hypervisor on AMD64 > > > > xen-qemu-dm-4.0 4.0.1-1 Xen > > Qemu Device Model virtual machine hardware emulator > > > > xen-tools 4.2-1 Tools to > > manage Xen virtual servers > > > > xen-utils-4.0 4.0.1-1 XEN > > administrative tools > > > > xen-utils-common 4.0.0-1 XEN > > administrative tools - common files > > > > xenstore-utils 4.0.1-1 > > Xenstore utilities for Xen > > > > xenwatch 0.5.4-2 > > Virtualization utilities, mostly for Xen > > > > > > > > ~# xm info > > > > host : dom0 > > > > release : 2.6.32-5-xen-amd64 > > > > version : #1 SMP Wed Oct 20 02:22:18 UTC 2010 > > > > machine : x86_64 > > > > nr_cpus : 8 > > > > nr_nodes : 2 > > > > cores_per_socket : 4 > > > > threads_per_core : 1 > > > > cpu_mhz : 2000 > > > > hw_caps : > > bfebfbff:28100800:00000000:00001b40:009ce3bd:00000000:00000001:00000000 > > > > virt_caps : hvm hvm_directio > > > > total_memory : 12279 > > > > free_memory : 3934 > > > > node_to_cpu : node0:0-3 > > > > node1:4-7 > > > > node_to_memory : node0:1996 > > > > node1:1938 > > > > node_to_dma32_mem : node0:1995 > > > > node1:0 > > > > max_node_id : 1 > > > > xen_major : 4 > > > > xen_minor : 0 > > > > xen_extra : .1 > > > > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 > > hvm-3.0-x 86_32p hvm-3.0-x86_64 > > > > xen_scheduler : credit > > > > xen_pagesize : 4096 > > > > platform_params : virt_start=0xffff800000000000 > > > > xen_changeset : unavailable > > > > xen_commandline : dom0_mem=2048M dom0_max_vcpus=2 loglvl=all > > guest_loglvl=all console=tty0 > > > > cc_compiler : gcc version 4.4.5 20100824 (prerelease) > > (Debian 4.4.4-11) > > > > cc_compile_by : waldi > > > > cc_compile_domain : debian.org > > > > cc_compile_date : Fri Sep 3 15:38:12 UTC 2010 > > > > xend_config_format : 4 > > > > > > > > thanks > > > > jf > > > > -- > > > > <° >< Jean-François Malouin McConnell Brain Imaging Centre > > > > Systems/Network Administrator Montréal Neurological Institute > > > > 3801 Rue University, Suite WB219 Montréal, Québec, H3A 2B4 > > > > Phone: 514-398-8924 Fax: 514-398-8948 > > > > > > > > _______________________________________________ > > > > Xen-users mailing list > > > > Xen-users@lists.xensource.com > > > > http://lists.xensource.com/xen-users > > > > > > _______________________________________________ > > > Xen-users mailing list > > > Xen-users@lists.xensource.com > > > http://lists.xensource.com/xen-users > > > > -- > > <° >< Jean-François Malouin McConnell Brain Imaging Centre > > Systems/Network Administrator Montréal Neurological Institute > > 3801 Rue University, Suite WB219 Montréal, Québec, H3A 2B4 > > Phone: 514-398-8924 Fax: 514-398-8948 > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > >> _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Maybe Matching Threads
- Icecast + Google Analytics
- Server renew...
- Xen PV DomU running Kernel 4.14.5-1.el7.elrepo.x86_64: xl -v vcpu-set <domU> <val> triggers domU kernel WARNING, then domU becomes unresponsive
- Xen PV DomU running Kernel 4.14.5-1.el7.elrepo.x86_64: xl -v vcpu-set <domU> <val> triggers domU kernel WARNING, then domU becomes unresponsive
- Strange R behavior for product of two sum of integers