Pasi Kärkkäinen
2010-Jan-28 21:25 UTC
[Xen-devel] Test results for save/restore with upstream pv_ops domU kernels
Hello, I just tried some save/restore tests with Fedora 12 Linux 2.6.31.12 kernels. The exact Fedora kernel versions are: 2.6.31.12-174.2.3.fc12.i686.PAE and 2.6.31.12-174.2.3.fc12.x86_64. Dom0 for these tests was CentOS 5.4 (Xen 3.1.2). - F12 32bit 1vcpu PV guest: save+restore OK, BUG() in guest dmesg after restore [1] - F12 64bit 1vcpu PV guest: save+restore OK, BUG() in guest dmesg after restore [2] - F12 32bit 2vcpu PV guest: save doesn''t work, guest stays as "migrating-f12test32" in "xm list" forever and has to be "xm destroy"ed. - F12 64bit 2vcpu PV guest: save doesn''t work, guest stays as "migrating-f12test64" in "xm list" forever and has to be "xm destroy"ed. What''s the best way to debug failing "xm save" ? There was no errors in "xm log", or in "xm dmesg". Also the guest "xm console" doesn''t show anything before it dies. Is it possible some of the save/restore related patches didn''t make it to 2.6.31.x stable kernels? BUG()s after "xm restore": [1] F12 32b 1vcpu: suspending xenstore... BUG: sleeping function called from invalid context at kernel/mutex.c:94 in_atomic(): 0, irqs_disabled(): 1, pid: 1136, name: kstop/0 Pid: 1136, comm: kstop/0 Tainted: G W 2.6.31.12-174.2.3.fc12.i686.PAE #1 Call Trace: [<c042f3fa>] __might_sleep+0xc6/0xcb [<c077a276>] mutex_lock+0x1d/0x3c [<c06404a3>] dpm_resume_noirq+0x17/0xe1 [<c060959d>] xen_suspend+0xc4/0xcf [<c046f96c>] stop_cpu+0x75/0xb4 [<c044d440>] worker_thread+0x13c/0x1bc [<c046f8f7>] ? stop_cpu+0x0/0xb4 [<c0450db9>] ? autoremove_wake_function+0x0/0x34 [<c044d304>] ? worker_thread+0x0/0x1bc [<c0450b0f>] kthread+0x70/0x75 [<c0450a9f>] ? kthread+0x0/0x75 [<c0409c07>] kernel_thread_helper+0x7/0x10 eth0: no IPv6 routers present [2] F12 64b 1vcpu: suspending xenstore... trying to map vcpu_info 0 at ffffc9000000b020, mfn 1d9997, offset 32 cpu 0 using vcpu_info at ffffc9000000b020 BUG: sleeping function called from invalid context at kernel/mutex.c:94 in_atomic(): 0, irqs_disabled(): 1, pid: 1112, name: kstop/0 Pid: 1112, comm: kstop/0 Tainted: G W 2.6.31.12-174.2.3.fc12.x86_64 #1 Call Trace: [<ffffffff8104026a>] __might_sleep+0xe6/0xe8 [<ffffffff8141c844>] mutex_lock+0x22/0x4e [<ffffffff812b266a>] dpm_resume_noirq+0x21/0x11f [<ffffffff812752a9>] xen_suspend+0xca/0xd1 [<ffffffff8108c33e>] stop_cpu+0x8c/0xd2 [<ffffffff810635a0>] worker_thread+0x18a/0x224 [<ffffffff81067b37>] ? autoremove_wake_function+0x0/0x39 [<ffffffff8141d6e9>] ? _spin_unlock_irqrestore+0x19/0x1b [<ffffffff81063416>] ? worker_thread+0x0/0x224 [<ffffffff810677b5>] kthread+0x91/0x99 [<ffffffff81012daa>] child_rip+0xa/0x20 [<ffffffff81011f97>] ? int_ret_from_sys_call+0x7/0x1b [<ffffffff8101271d>] ? retint_restore_args+0x5/0x6 [<ffffffff81012da0>] ? child_rip+0x0/0x20 -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2010-Jan-29 10:35 UTC
[Xen-devel] Re: Test results for save/restore with upstream pv_ops domU kernels
On Thu, 2010-01-28 at 21:25 +0000, Pasi Kärkkäinen wrote:> Hello, > > I just tried some save/restore tests with Fedora 12 Linux 2.6.31.12 kernels. > The exact Fedora kernel versions are: 2.6.31.12-174.2.3.fc12.i686.PAE and 2.6.31.12-174.2.3.fc12.x86_64. > > Dom0 for these tests was CentOS 5.4 (Xen 3.1.2). > > - F12 32bit 1vcpu PV guest: > save+restore OK, BUG() in guest dmesg after restore [1] > > - F12 64bit 1vcpu PV guest: > save+restore OK, BUG() in guest dmesg after restore [2]I think those are the same underlying bug and are fixed by http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=777df2b017ef34b2d1a172bf769582158839a860> > - F12 32bit 2vcpu PV guest: > save doesn''t work, guest stays as "migrating-f12test32" in "xm list" forever and has to be "xm destroy"ed. > > - F12 64bit 2vcpu PV guest: > save doesn''t work, guest stays as "migrating-f12test64" in "xm list" forever and has to be "xm destroy"ed. > > > What''s the best way to debug failing "xm save" ? There was no errors in "xm log", or in "xm dmesg".I think you might see some stuff in /var/log/xen/something but I don''t have any particular tips apart from "add printf/printk".> Also the guest "xm console" doesn''t show anything before it dies. > > Is it possible some of the save/restore related patches didn''t make it to 2.6.31.x stable kernels?AFAIK they only went into the 2.6.32 stable branch. Unfortunately I think the 2.6.31 stable series has come to an end now. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Jan-29 10:53 UTC
[Xen-devel] Re: Test results for save/restore with upstream pv_ops domU kernels
On Fri, Jan 29, 2010 at 10:35:32AM +0000, Ian Campbell wrote:> On Thu, 2010-01-28 at 21:25 +0000, Pasi Kärkkäinen wrote: > > Hello, > > > > I just tried some save/restore tests with Fedora 12 Linux 2.6.31.12 kernels. > > The exact Fedora kernel versions are: 2.6.31.12-174.2.3.fc12.i686.PAE and 2.6.31.12-174.2.3.fc12.x86_64. > > > > Dom0 for these tests was CentOS 5.4 (Xen 3.1.2). > > > > - F12 32bit 1vcpu PV guest: > > save+restore OK, BUG() in guest dmesg after restore [1] > > > > - F12 64bit 1vcpu PV guest: > > save+restore OK, BUG() in guest dmesg after restore [2] > > I think those are the same underlying bug and are fixed by > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=777df2b017ef34b2d1a172bf769582158839a860 >Ok. There was also this traceback in the beginning of boot, on all 32bit/64bit, 1vcpu2/vcpu guest combinations: Performance Counters: Core2 events, Intel PMU driver. ------------[ cut here ]------------ WARNING: at arch/x86/kernel/apic/apic.c:247 native_apic_write_dummy+0x32/0x3e() (Not tainted) Modules linked in: Pid: 0, comm: swapper Not tainted 2.6.31.12-174.2.3.fc12.i686.PAE #1 Call Trace: [<c043db4b>] warn_slowpath_common+0x70/0x87 [<c041cfb2>] ? native_apic_write_dummy+0x32/0x3e [<c043db74>] warn_slowpath_null+0x12/0x15 [<c041cfb2>] native_apic_write_dummy+0x32/0x3e [<c0411e04>] perf_counters_lapic_init+0x30/0x32 [<c09b3b1b>] init_hw_perf_counters+0x2bc/0x355 [<c09b3628>] identify_boot_cpu+0x21/0x23 [<c09b378e>] check_bugs+0xb/0xdc [<c047fd73>] ? delayacct_init+0x47/0x4c [<c09ab8b4>] start_kernel+0x31c/0x330 [<c09ab081>] i386_start_kernel+0x70/0x77 [<c09ae2bb>] xen_start_kernel+0x4b9/0x4c1 [<c04090a1>] ? syscall_exit+0x1/0x16 ---[ end trace a7919e7f17c0a725 ]--- Full boot logs here: http://pasik.reaktio.net/xen/debug/fedora/> > > > - F12 32bit 2vcpu PV guest: > > save doesn''t work, guest stays as "migrating-f12test32" in "xm list" forever and has to be "xm destroy"ed. > > > > - F12 64bit 2vcpu PV guest: > > save doesn''t work, guest stays as "migrating-f12test64" in "xm list" forever and has to be "xm destroy"ed. > > > > > > What''s the best way to debug failing "xm save" ? There was no errors in "xm log", or in "xm dmesg". > > I think you might see some stuff in /var/log/xen/something but I don''t > have any particular tips apart from "add printf/printk". >I''ll check /var/log/xen/.> > Also the guest "xm console" doesn''t show anything before it dies. > > > > Is it possible some of the save/restore related patches didn''t make it to 2.6.31.x stable kernels? > > AFAIK they only went into the 2.6.32 stable branch. Unfortunately I > think the 2.6.31 stable series has come to an end now. >Ok. I''ll test 2.6.32.latest aswell. -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Feb-01 15:10 UTC
Re: [Xen-devel] Re: Test results for save/restore with upstream pv_ops domU kernels, 2.6.32.7 works OK
On Fri, Jan 29, 2010 at 12:53:38PM +0200, Pasi Kärkkäinen wrote:> On Fri, Jan 29, 2010 at 10:35:32AM +0000, Ian Campbell wrote: > > On Thu, 2010-01-28 at 21:25 +0000, Pasi Kärkkäinen wrote: > > > Hello, > > > > > > I just tried some save/restore tests with Fedora 12 Linux 2.6.31.12 kernels. > > > The exact Fedora kernel versions are: 2.6.31.12-174.2.3.fc12.i686.PAE and 2.6.31.12-174.2.3.fc12.x86_64. > > > > > > Dom0 for these tests was CentOS 5.4 (Xen 3.1.2). > > > > > > - F12 32bit 1vcpu PV guest: > > > save+restore OK, BUG() in guest dmesg after restore [1] > > > > > > - F12 64bit 1vcpu PV guest: > > > save+restore OK, BUG() in guest dmesg after restore [2] > > > > I think those are the same underlying bug and are fixed by > > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=777df2b017ef34b2d1a172bf769582158839a860 > > > > Ok. > > There was also this traceback in the beginning of boot, on all 32bit/64bit, 1vcpu2/vcpu guest combinations: > > Performance Counters: Core2 events, Intel PMU driver. > ------------[ cut here ]------------ > WARNING: at arch/x86/kernel/apic/apic.c:247 native_apic_write_dummy+0x32/0x3e() (Not tainted) > Modules linked in: > Pid: 0, comm: swapper Not tainted 2.6.31.12-174.2.3.fc12.i686.PAE #1 > Call Trace: > [<c043db4b>] warn_slowpath_common+0x70/0x87 > [<c041cfb2>] ? native_apic_write_dummy+0x32/0x3e > [<c043db74>] warn_slowpath_null+0x12/0x15 > [<c041cfb2>] native_apic_write_dummy+0x32/0x3e > [<c0411e04>] perf_counters_lapic_init+0x30/0x32 > [<c09b3b1b>] init_hw_perf_counters+0x2bc/0x355 > [<c09b3628>] identify_boot_cpu+0x21/0x23 > [<c09b378e>] check_bugs+0xb/0xdc > [<c047fd73>] ? delayacct_init+0x47/0x4c > [<c09ab8b4>] start_kernel+0x31c/0x330 > [<c09ab081>] i386_start_kernel+0x70/0x77 > [<c09ae2bb>] xen_start_kernel+0x4b9/0x4c1 > [<c04090a1>] ? syscall_exit+0x1/0x16 > ---[ end trace a7919e7f17c0a725 ]--- > > Full boot logs here: > http://pasik.reaktio.net/xen/debug/fedora/ >This boot-time traceback disappeared when I updated the guest to 2.6.32.7.> > > > > > > - F12 32bit 2vcpu PV guest: > > > save doesn''t work, guest stays as "migrating-f12test32" in "xm list" forever and has to be "xm destroy"ed. > > > > > > - F12 64bit 2vcpu PV guest: > > > save doesn''t work, guest stays as "migrating-f12test64" in "xm list" forever and has to be "xm destroy"ed. > > > > > > > > > What''s the best way to debug failing "xm save" ? There was no errors in "xm log", or in "xm dmesg". > > > > I think you might see some stuff in /var/log/xen/something but I don''t > > have any particular tips apart from "add printf/printk". > > > > I''ll check /var/log/xen/. > > > > Also the guest "xm console" doesn''t show anything before it dies. > > > > > > Is it possible some of the save/restore related patches didn''t make it to 2.6.31.x stable kernels? > > > > AFAIK they only went into the 2.6.32 stable branch. Unfortunately I > > think the 2.6.31 stable series has come to an end now. > > > > Ok. I''ll test 2.6.32.latest aswell. >I grabbed upstream kernel.org Linux 2.6.32.7, and tested the following combinations: - F12 32bit 1vcpu PV guest - F12 32bit 2vcpu PV guest - F12 64bit 1vcpu PV guest - F12 64bit 2vcpu PV guest save+restore was successfull for all of the above guests running 2.6.32.7. No BUGs or tracebacks anymore. Any tips for git magic to get all the recent save/restore fixes that went to 2.6.32.x, so I could send them to Fedora people to apply to F12 kernel? Thanks! -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andrew Jones
2010-Feb-01 15:26 UTC
Re: [Xen-devel] Re: Test results for save/restore with upstream pv_ops domU kernels, 2.6.32.7 works OK
On 02/01/2010 04:10 PM, Pasi Kärkkäinen wrote:> On Fri, Jan 29, 2010 at 12:53:38PM +0200, Pasi Kärkkäinen wrote: >> On Fri, Jan 29, 2010 at 10:35:32AM +0000, Ian Campbell wrote: >>> On Thu, 2010-01-28 at 21:25 +0000, Pasi Kärkkäinen wrote: >>>> Hello, >>>> >>>> I just tried some save/restore tests with Fedora 12 Linux 2.6.31.12 kernels. >>>> The exact Fedora kernel versions are: 2.6.31.12-174.2.3.fc12.i686.PAE and 2.6.31.12-174.2.3.fc12.x86_64. >>>> >>>> Dom0 for these tests was CentOS 5.4 (Xen 3.1.2). >>>> >>>> - F12 32bit 1vcpu PV guest: >>>> save+restore OK, BUG() in guest dmesg after restore [1] >>>> >>>> - F12 64bit 1vcpu PV guest: >>>> save+restore OK, BUG() in guest dmesg after restore [2] >>> >>> I think those are the same underlying bug and are fixed by >>> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=777df2b017ef34b2d1a172bf769582158839a860 >>> >> >> Ok. >> >> There was also this traceback in the beginning of boot, on all 32bit/64bit, 1vcpu2/vcpu guest combinations: >> >> Performance Counters: Core2 events, Intel PMU driver. >> ------------[ cut here ]------------ >> WARNING: at arch/x86/kernel/apic/apic.c:247 native_apic_write_dummy+0x32/0x3e() (Not tainted) >> Modules linked in: >> Pid: 0, comm: swapper Not tainted 2.6.31.12-174.2.3.fc12.i686.PAE #1 >> Call Trace: >> [<c043db4b>] warn_slowpath_common+0x70/0x87 >> [<c041cfb2>] ? native_apic_write_dummy+0x32/0x3e >> [<c043db74>] warn_slowpath_null+0x12/0x15 >> [<c041cfb2>] native_apic_write_dummy+0x32/0x3e >> [<c0411e04>] perf_counters_lapic_init+0x30/0x32 >> [<c09b3b1b>] init_hw_perf_counters+0x2bc/0x355 >> [<c09b3628>] identify_boot_cpu+0x21/0x23 >> [<c09b378e>] check_bugs+0xb/0xdc >> [<c047fd73>] ? delayacct_init+0x47/0x4c >> [<c09ab8b4>] start_kernel+0x31c/0x330 >> [<c09ab081>] i386_start_kernel+0x70/0x77 >> [<c09ae2bb>] xen_start_kernel+0x4b9/0x4c1 >> [<c04090a1>] ? syscall_exit+0x1/0x16 >> ---[ end trace a7919e7f17c0a725 ]--- >> >> Full boot logs here: >> http://pasik.reaktio.net/xen/debug/fedora/ >> > > > This boot-time traceback disappeared when I updated the guest to 2.6.32.7. > > >> >>>> >>>> - F12 32bit 2vcpu PV guest: >>>> save doesn''t work, guest stays as "migrating-f12test32" in "xm list" forever and has to be "xm destroy"ed. >>>> >>>> - F12 64bit 2vcpu PV guest: >>>> save doesn''t work, guest stays as "migrating-f12test64" in "xm list" forever and has to be "xm destroy"ed. >>>> >>>> >>>> What''s the best way to debug failing "xm save" ? There was no errors in "xm log", or in "xm dmesg". >>> >>> I think you might see some stuff in /var/log/xen/something but I don''t >>> have any particular tips apart from "add printf/printk". >>> >> >> I''ll check /var/log/xen/. >> >>>> Also the guest "xm console" doesn''t show anything before it dies. >>>> >>>> Is it possible some of the save/restore related patches didn''t make it to 2.6.31.x stable kernels? >>> >>> AFAIK they only went into the 2.6.32 stable branch. Unfortunately I >>> think the 2.6.31 stable series has come to an end now. >>> >> >> Ok. I''ll test 2.6.32.latest aswell. >> > > I grabbed upstream kernel.org Linux 2.6.32.7, and tested the following combinations: > > - F12 32bit 1vcpu PV guest > - F12 32bit 2vcpu PV guest > - F12 64bit 1vcpu PV guest > - F12 64bit 2vcpu PV guest > > save+restore was successfull for all of the above guests running 2.6.32.7. > No BUGs or tracebacks anymore. > > Any tips for git magic to get all the recent save/restore fixes that went to 2.6.32.x, > so I could send them to Fedora people to apply to F12 kernel? >This is great news, but it might a good idea to try 2 or more save-restore rounds in row first, if you haven''t already. In the past I''ve seen 1 save/restore work, but then the 2nd round fail. Although, usually there''s some symptom of badness on the 1st round as well. Thanks, Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Feb-01 16:12 UTC
Re: [Xen-devel] Re: Test results for save/restore with upstream pv_ops domU kernels, 2.6.32.7 works OK
On Mon, Feb 01, 2010 at 04:26:05PM +0100, Andrew Jones wrote:> On 02/01/2010 04:10 PM, Pasi Kärkkäinen wrote: > > On Fri, Jan 29, 2010 at 12:53:38PM +0200, Pasi Kärkkäinen wrote: > >> On Fri, Jan 29, 2010 at 10:35:32AM +0000, Ian Campbell wrote: > >>> On Thu, 2010-01-28 at 21:25 +0000, Pasi Kärkkäinen wrote: > >>>> Hello, > >>>> > >>>> I just tried some save/restore tests with Fedora 12 Linux 2.6.31.12 kernels. > >>>> The exact Fedora kernel versions are: 2.6.31.12-174.2.3.fc12.i686.PAE and 2.6.31.12-174.2.3.fc12.x86_64. > >>>> > >>>> Dom0 for these tests was CentOS 5.4 (Xen 3.1.2). > >>>> > >>>> - F12 32bit 1vcpu PV guest: > >>>> save+restore OK, BUG() in guest dmesg after restore [1] > >>>> > >>>> - F12 64bit 1vcpu PV guest: > >>>> save+restore OK, BUG() in guest dmesg after restore [2] > >>> > >>> I think those are the same underlying bug and are fixed by > >>> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=777df2b017ef34b2d1a172bf769582158839a860 > >>> > >> > >> Ok. > >> > >> There was also this traceback in the beginning of boot, on all 32bit/64bit, 1vcpu2/vcpu guest combinations: > >> > >> Performance Counters: Core2 events, Intel PMU driver. > >> ------------[ cut here ]------------ > >> WARNING: at arch/x86/kernel/apic/apic.c:247 native_apic_write_dummy+0x32/0x3e() (Not tainted) > >> Modules linked in: > >> Pid: 0, comm: swapper Not tainted 2.6.31.12-174.2.3.fc12.i686.PAE #1 > >> Call Trace: > >> [<c043db4b>] warn_slowpath_common+0x70/0x87 > >> [<c041cfb2>] ? native_apic_write_dummy+0x32/0x3e > >> [<c043db74>] warn_slowpath_null+0x12/0x15 > >> [<c041cfb2>] native_apic_write_dummy+0x32/0x3e > >> [<c0411e04>] perf_counters_lapic_init+0x30/0x32 > >> [<c09b3b1b>] init_hw_perf_counters+0x2bc/0x355 > >> [<c09b3628>] identify_boot_cpu+0x21/0x23 > >> [<c09b378e>] check_bugs+0xb/0xdc > >> [<c047fd73>] ? delayacct_init+0x47/0x4c > >> [<c09ab8b4>] start_kernel+0x31c/0x330 > >> [<c09ab081>] i386_start_kernel+0x70/0x77 > >> [<c09ae2bb>] xen_start_kernel+0x4b9/0x4c1 > >> [<c04090a1>] ? syscall_exit+0x1/0x16 > >> ---[ end trace a7919e7f17c0a725 ]--- > >> > >> Full boot logs here: > >> http://pasik.reaktio.net/xen/debug/fedora/ > >> > > > > > > This boot-time traceback disappeared when I updated the guest to 2.6.32.7. > > > > > >> > >>>> > >>>> - F12 32bit 2vcpu PV guest: > >>>> save doesn''t work, guest stays as "migrating-f12test32" in "xm list" forever and has to be "xm destroy"ed. > >>>> > >>>> - F12 64bit 2vcpu PV guest: > >>>> save doesn''t work, guest stays as "migrating-f12test64" in "xm list" forever and has to be "xm destroy"ed. > >>>> > >>>> > >>>> What''s the best way to debug failing "xm save" ? There was no errors in "xm log", or in "xm dmesg". > >>> > >>> I think you might see some stuff in /var/log/xen/something but I don''t > >>> have any particular tips apart from "add printf/printk". > >>> > >> > >> I''ll check /var/log/xen/. > >> > >>>> Also the guest "xm console" doesn''t show anything before it dies. > >>>> > >>>> Is it possible some of the save/restore related patches didn''t make it to 2.6.31.x stable kernels? > >>> > >>> AFAIK they only went into the 2.6.32 stable branch. Unfortunately I > >>> think the 2.6.31 stable series has come to an end now. > >>> > >> > >> Ok. I''ll test 2.6.32.latest aswell. > >> > > > > I grabbed upstream kernel.org Linux 2.6.32.7, and tested the following combinations: > > > > - F12 32bit 1vcpu PV guest > > - F12 32bit 2vcpu PV guest > > - F12 64bit 1vcpu PV guest > > - F12 64bit 2vcpu PV guest > > > > save+restore was successfull for all of the above guests running 2.6.32.7. > > No BUGs or tracebacks anymore. > > > > Any tips for git magic to get all the recent save/restore fixes that went to 2.6.32.x, > > so I could send them to Fedora people to apply to F12 kernel? > > > > This is great news, but it might a good idea to try 2 or more > save-restore rounds in row first, if you haven''t already. In the past > I''ve seen 1 save/restore work, but then the 2nd round fail. Although, > usually there''s some symptom of badness on the 1st round as well. >I forgot to mention that I tried twice with all of the above guests :) It seems stable. I just did one more test.. save+restore 5 times in a row, with 4 vcpu PV guest. No problems found. -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Feb-25 15:14 UTC
[Xen-devel] Re: Test results for save/restore with upstream pv_ops domU kernels, 2.6.33 problems
On Fri, Jan 29, 2010 at 10:35:32AM +0000, Ian Campbell wrote:> On Thu, 2010-01-28 at 21:25 +0000, Pasi Kärkkäinen wrote: > > Hello, > > > > I just tried some save/restore tests with Fedora 12 Linux 2.6.31.12 kernels. > > The exact Fedora kernel versions are: 2.6.31.12-174.2.3.fc12.i686.PAE and 2.6.31.12-174.2.3.fc12.x86_64. > > > > Dom0 for these tests was CentOS 5.4 (Xen 3.1.2). > > > > - F12 32bit 1vcpu PV guest: > > save+restore OK, BUG() in guest dmesg after restore [1] > > > > - F12 64bit 1vcpu PV guest: > > save+restore OK, BUG() in guest dmesg after restore [2] > > I think those are the same underlying bug and are fixed by > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=777df2b017ef34b2d1a172bf769582158839a860 > > > > > - F12 32bit 2vcpu PV guest: > > save doesn''t work, guest stays as "migrating-f12test32" in "xm list" forever and has to be "xm destroy"ed. > > > > - F12 64bit 2vcpu PV guest: > > save doesn''t work, guest stays as "migrating-f12test64" in "xm list" forever and has to be "xm destroy"ed. > > > > > > What''s the best way to debug failing "xm save" ? There was no errors in "xm log", or in "xm dmesg". > > I think you might see some stuff in /var/log/xen/something but I don''t > have any particular tips apart from "add printf/printk". > > > Also the guest "xm console" doesn''t show anything before it dies. > > > > Is it possible some of the save/restore related patches didn''t make it to 2.6.31.x stable kernels? > > AFAIK they only went into the 2.6.32 stable branch. Unfortunately I > think the 2.6.31 stable series has come to an end now. >Hello, An user on ##xen just reported that 2.6.32.9 save/restore works OK, but 2.6.33 fails/crashes. Some of the patches that went to 2.6.32.x are not in 2.6.33? -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2010-Feb-25 15:17 UTC
[Xen-devel] Re: Test results for save/restore with upstream pv_ops domU kernels, 2.6.33 problems
On Thu, 2010-02-25 at 15:14 +0000, Pasi Kärkkäinen wrote:> On Fri, Jan 29, 2010 at 10:35:32AM +0000, Ian Campbell wrote: > > On Thu, 2010-01-28 at 21:25 +0000, Pasi Kärkkäinen wrote: > > > Hello, > > > > > > I just tried some save/restore tests with Fedora 12 Linux 2.6.31.12 kernels. > > > The exact Fedora kernel versions are: 2.6.31.12-174.2.3.fc12.i686.PAE and 2.6.31.12-174.2.3.fc12.x86_64. > > > > > > Dom0 for these tests was CentOS 5.4 (Xen 3.1.2). > > > > > > - F12 32bit 1vcpu PV guest: > > > save+restore OK, BUG() in guest dmesg after restore [1] > > > > > > - F12 64bit 1vcpu PV guest: > > > save+restore OK, BUG() in guest dmesg after restore [2] > > > > I think those are the same underlying bug and are fixed by > > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=777df2b017ef34b2d1a172bf769582158839a860 > > > > > > > > - F12 32bit 2vcpu PV guest: > > > save doesn''t work, guest stays as "migrating-f12test32" in "xm list" forever and has to be "xm destroy"ed. > > > > > > - F12 64bit 2vcpu PV guest: > > > save doesn''t work, guest stays as "migrating-f12test64" in "xm list" forever and has to be "xm destroy"ed. > > > > > > > > > What''s the best way to debug failing "xm save" ? There was no errors in "xm log", or in "xm dmesg". > > > > I think you might see some stuff in /var/log/xen/something but I don''t > > have any particular tips apart from "add printf/printk". > > > > > Also the guest "xm console" doesn''t show anything before it dies. > > > > > > Is it possible some of the save/restore related patches didn''t make it to 2.6.31.x stable kernels? > > > > AFAIK they only went into the 2.6.32 stable branch. Unfortunately I > > think the 2.6.31 stable series has come to an end now. > > > > Hello, > > An user on ##xen just reported that 2.6.32.9 save/restore works OK, > but 2.6.33 fails/crashes. > > Some of the patches that went to 2.6.32.x are not in 2.6.33?I don''t think so -- the acceptance criteria for a stable tree is that the patch is in mainline already. Must be some new breakage. Ian.> > -- Pasi >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Feb-25 15:27 UTC
[Xen-devel] Re: Test results for save/restore with upstream pv_ops domU kernels, 2.6.33 problems
On Thu, Feb 25, 2010 at 03:17:08PM +0000, Ian Campbell wrote:> On Thu, 2010-02-25 at 15:14 +0000, Pasi Kärkkäinen wrote: > > On Fri, Jan 29, 2010 at 10:35:32AM +0000, Ian Campbell wrote: > > > On Thu, 2010-01-28 at 21:25 +0000, Pasi Kärkkäinen wrote: > > > > Hello, > > > > > > > > I just tried some save/restore tests with Fedora 12 Linux 2.6.31.12 kernels. > > > > The exact Fedora kernel versions are: 2.6.31.12-174.2.3.fc12.i686.PAE and 2.6.31.12-174.2.3.fc12.x86_64. > > > > > > > > Dom0 for these tests was CentOS 5.4 (Xen 3.1.2). > > > > > > > > - F12 32bit 1vcpu PV guest: > > > > save+restore OK, BUG() in guest dmesg after restore [1] > > > > > > > > - F12 64bit 1vcpu PV guest: > > > > save+restore OK, BUG() in guest dmesg after restore [2] > > > > > > I think those are the same underlying bug and are fixed by > > > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=777df2b017ef34b2d1a172bf769582158839a860 > > > > > > > > > > > - F12 32bit 2vcpu PV guest: > > > > save doesn''t work, guest stays as "migrating-f12test32" in "xm list" forever and has to be "xm destroy"ed. > > > > > > > > - F12 64bit 2vcpu PV guest: > > > > save doesn''t work, guest stays as "migrating-f12test64" in "xm list" forever and has to be "xm destroy"ed. > > > > > > > > > > > > What''s the best way to debug failing "xm save" ? There was no errors in "xm log", or in "xm dmesg". > > > > > > I think you might see some stuff in /var/log/xen/something but I don''t > > > have any particular tips apart from "add printf/printk". > > > > > > > Also the guest "xm console" doesn''t show anything before it dies. > > > > > > > > Is it possible some of the save/restore related patches didn''t make it to 2.6.31.x stable kernels? > > > > > > AFAIK they only went into the 2.6.32 stable branch. Unfortunately I > > > think the 2.6.31 stable series has come to an end now. > > > > > > > Hello, > > > > An user on ##xen just reported that 2.6.32.9 save/restore works OK, > > but 2.6.33 fails/crashes. > > > > Some of the patches that went to 2.6.32.x are not in 2.6.33? > > I don''t think so -- the acceptance criteria for a stable tree is that > the patch is in mainline already. Must be some new breakage. >Ok.. I''ll do some testing myself. -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Feb-25 20:02 UTC
[Xen-devel] Re: Test results for save/restore with upstream pv_ops domU kernels, 2.6.33 problems
On Thu, Feb 25, 2010 at 05:27:13PM +0200, Pasi Kärkkäinen wrote:> > > > > > > > > > Is it possible some of the save/restore related patches didn''t make it to 2.6.31.x stable kernels? > > > > > > > > AFAIK they only went into the 2.6.32 stable branch. Unfortunately I > > > > think the 2.6.31 stable series has come to an end now. > > > > > > > > > > Hello, > > > > > > An user on ##xen just reported that 2.6.32.9 save/restore works OK, > > > but 2.6.33 fails/crashes. > > > > > > Some of the patches that went to 2.6.32.x are not in 2.6.33? > > > > I don''t think so -- the acceptance criteria for a stable tree is that > > the patch is in mainline already. Must be some new breakage. > > > > Ok.. I''ll do some testing myself. >I had to patch 2.6.33 to get it boot without crashing, (Jeremy sent me a patch to fix it), but after that save/restore worked OK for me. I tried save+restore 10 times in a row for a x86_64 4-vcpu 2.6.33 guest with 2 GB of memory, no problems found on my environment (EL 5.4 dom0). -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel