Hi, I tested the latest xen-unstable (changeset: 20128) and the latest linux-2.6.18-xen (changeset: 931). A guest OS is booted up slower than before. And the following messages are shown by the guest OS. Switching to new root and running init. unmounting old /dev unmounting old /proc unmounting old /sys *************************************************************** *************************************************************** ** WARNING: Currently emulating unsupported memory accesses ** ** in /lib/tls glibc libraries. The emulation is ** ** slow. To ensure full performance you should ** ** install a ''xen-friendly'' (nosegneg) version of ** ** the library, or disable tls support by executing ** ** the following as root: ** ** mv /lib/tls /lib/tls.disabled ** ** Offending process: modprobe (pid=761) ** *************************************************************** *************************************************************** Pausing... 5<3>BUG: soft lockup detected on CPU#0! [<c0151205>] softlockup_tick+0xa5/0xd0 [<c010978a>] timer_interrupt+0x2fa/0x6c0 [<c011d8c1>] __activate_task+0x21/0x40 [<c012fed0>] lock_timer_base+0x20/0x50 [<c0151563>] handle_IRQ_event+0x33/0xa0 [<c0151678>] __do_IRQ+0xa8/0x120 [<c01076e1>] do_IRQ+0x31/0x80 [<c02af6b0>] neigh_periodic_timer+0x0/0x140 [<c024ced5>] evtchn_do_upcall+0xe5/0x1f0 [<c0115fb0>] do_fixup_4gb_segment+0x0/0x170 [<c0105ba3>] hypervisor_callback+0x33/0x3b [<c0115fb0>] do_fixup_4gb_segment+0x0/0x170 [<c01ed68b>] delay_tsc+0xb/0x20 [<c01ed6d6>] __delay+0x6/0x10 [<c01160f0>] do_fixup_4gb_segment+0x140/0x170 [<c0169597>] do_munmap+0x197/0x200 [<c01ee0a4>] copy_to_user+0x34/0x70 [<c0105b6b>] error_code+0x2b/0x30 Continuing... FYI, when I reverted changeset 20122 of xen-unstable, the guest OS is booted up as before. Best regards, Kan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Maskai, Thanks for the report. It''s a good idea to Cc the patch author when making bug reports. In this case it''s George Dunlap who I''ve cc''ed in this reply. -- Keir On 31/08/2009 07:50, "Masaki Kanno" <kanno.masaki@jp.fujitsu.com> wrote:> Hi, > > I tested the latest xen-unstable (changeset: 20128) and the latest > linux-2.6.18-xen (changeset: 931). A guest OS is booted up slower > than before. And the following messages are shown by the guest OS. > > Switching to new root and running init. > unmounting old /dev > unmounting old /proc > unmounting old /sys > > *************************************************************** > *************************************************************** > ** WARNING: Currently emulating unsupported memory accesses ** > ** in /lib/tls glibc libraries. The emulation is ** > ** slow. To ensure full performance you should ** > ** install a ''xen-friendly'' (nosegneg) version of ** > ** the library, or disable tls support by executing ** > ** the following as root: ** > ** mv /lib/tls /lib/tls.disabled ** > ** Offending process: modprobe (pid=761) ** > *************************************************************** > *************************************************************** > > Pausing... 5<3>BUG: soft lockup detected on CPU#0! > [<c0151205>] softlockup_tick+0xa5/0xd0 > [<c010978a>] timer_interrupt+0x2fa/0x6c0 > [<c011d8c1>] __activate_task+0x21/0x40 > [<c012fed0>] lock_timer_base+0x20/0x50 > [<c0151563>] handle_IRQ_event+0x33/0xa0 > [<c0151678>] __do_IRQ+0xa8/0x120 > [<c01076e1>] do_IRQ+0x31/0x80 > [<c02af6b0>] neigh_periodic_timer+0x0/0x140 > [<c024ced5>] evtchn_do_upcall+0xe5/0x1f0 > [<c0115fb0>] do_fixup_4gb_segment+0x0/0x170 > [<c0105ba3>] hypervisor_callback+0x33/0x3b > [<c0115fb0>] do_fixup_4gb_segment+0x0/0x170 > [<c01ed68b>] delay_tsc+0xb/0x20 > [<c01ed6d6>] __delay+0x6/0x10 > [<c01160f0>] do_fixup_4gb_segment+0x140/0x170 > [<c0169597>] do_munmap+0x197/0x200 > [<c01ee0a4>] copy_to_user+0x34/0x70 > [<c0105b6b>] error_code+0x2b/0x30 > Continuing... > > > FYI, when I reverted changeset 20122 of xen-unstable, the guest OS > is booted up as before. > > Best regards, > Kan > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hmm, it appears my patch still debits credits for VMs that aren''t earning credits anymore; such VMs can earn an unlimited amount of negative credit before becoming active again. Try the attached patches; if it doesn''t fix the problem, please take a short trace during boot using the following command and send it to me: # xentrace -D -e 0x2f000 -S 128 -s 1000 /tmp/sched-boot.trace Thanks, -George On Mon, Aug 31, 2009 at 9:34 AM, Keir Fraser<keir.fraser@eu.citrix.com> wrote:> Hi Maskai, > > Thanks for the report. It''s a good idea to Cc the patch author when making > bug reports. In this case it''s George Dunlap who I''ve cc''ed in this reply. > > -- Keir > > On 31/08/2009 07:50, "Masaki Kanno" <kanno.masaki@jp.fujitsu.com> wrote: > >> Hi, >> >> I tested the latest xen-unstable (changeset: 20128) and the latest >> linux-2.6.18-xen (changeset: 931). A guest OS is booted up slower >> than before. And the following messages are shown by the guest OS. >> >> Switching to new root and running init. >> unmounting old /dev >> unmounting old /proc >> unmounting old /sys >> >> *************************************************************** >> *************************************************************** >> ** WARNING: Currently emulating unsupported memory accesses ** >> ** in /lib/tls glibc libraries. The emulation is ** >> ** slow. To ensure full performance you should ** >> ** install a ''xen-friendly'' (nosegneg) version of ** >> ** the library, or disable tls support by executing ** >> ** the following as root: ** >> ** mv /lib/tls /lib/tls.disabled ** >> ** Offending process: modprobe (pid=761) ** >> *************************************************************** >> *************************************************************** >> >> Pausing... 5<3>BUG: soft lockup detected on CPU#0! >> [<c0151205>] softlockup_tick+0xa5/0xd0 >> [<c010978a>] timer_interrupt+0x2fa/0x6c0 >> [<c011d8c1>] __activate_task+0x21/0x40 >> [<c012fed0>] lock_timer_base+0x20/0x50 >> [<c0151563>] handle_IRQ_event+0x33/0xa0 >> [<c0151678>] __do_IRQ+0xa8/0x120 >> [<c01076e1>] do_IRQ+0x31/0x80 >> [<c02af6b0>] neigh_periodic_timer+0x0/0x140 >> [<c024ced5>] evtchn_do_upcall+0xe5/0x1f0 >> [<c0115fb0>] do_fixup_4gb_segment+0x0/0x170 >> [<c0105ba3>] hypervisor_callback+0x33/0x3b >> [<c0115fb0>] do_fixup_4gb_segment+0x0/0x170 >> [<c01ed68b>] delay_tsc+0xb/0x20 >> [<c01ed6d6>] __delay+0x6/0x10 >> [<c01160f0>] do_fixup_4gb_segment+0x140/0x170 >> [<c0169597>] do_munmap+0x197/0x200 >> [<c01ee0a4>] copy_to_user+0x34/0x70 >> [<c0105b6b>] error_code+0x2b/0x30 >> Continuing... >> >> >> FYI, when I reverted changeset 20122 of xen-unstable, the guest OS >> is booted up as before. >> >> Best regards, >> Kan >> >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi George, I will try them on today. Kan Mon, 31 Aug 2009 14:10:47 +0100, George Dunlap wrote:>Hmm, it appears my patch still debits credits for VMs that aren''t >earning credits anymore; such VMs can earn an unlimited amount of >negative credit before becoming active again. > >Try the attached patches; if it doesn''t fix the problem, please take a >short trace during boot using the following command and send it to me: > ># xentrace -D -e 0x2f000 -S 128 -s 1000 /tmp/sched-boot.trace > >Thanks, > -George > >On Mon, Aug 31, 2009 at 9:34 AM, Keir Fraser<keir.fraser@eu.citrix.com> >wrote: >> Hi Maskai, >> >> Thanks for the report. It''s a good idea to Cc the patch author when making >> bug reports. In this case it''s George Dunlap who I''ve cc''ed in this reply. >> >> I -- Keir >> >> On 31/08/2009 07:50, "Masaki Kanno" <kanno.masaki@jp.fujitsu.com> wrote: >> >>> Hi, >>> >>> I tested the latest xen-unstable (changeset: 20128) and the latest >>> linux-2.6.18-xen (changeset: 931). I A guest OS is booted up slower >>> than before. I And the following messages are shown by the guest OS. >>> >>> Switching to new root and running init. >>> unmounting old /dev >>> unmounting old /proc >>> unmounting old /sys >>> >>> I *************************************************************** >>> I *************************************************************** >>> I ** WARNING: Currently emulating unsupported memory accesses I ** >>> I ** I I I I I in /lib/tls glibc libraries. The emulation is I I ** >>> I ** I I I I I slow. To ensure full performance you should I I I ** >>> I ** I I I I I install a ''xen-friendly'' (nosegneg) version of I ** >>> I ** I I I I I the library, or disable tls support by executing ** >>> I ** I I I I I the following as root: I I I I I I I I I I I I I ** >>> I ** I I I I I mv /lib/tls /lib/tls.disabled I I I I I I I I I I ** >>> I ** Offending process: modprobe (pid=761) I I I I I I I I I I ** >>> I *************************************************************** >>> I *************************************************************** >>> >>> Pausing... 5<3>BUG: soft lockup detected on CPU#0! >>> I [<c0151205>] softlockup_tick+0xa5/0xd0 >>> I [<c010978a>] timer_interrupt+0x2fa/0x6c0 >>> I [<c011d8c1>] __activate_task+0x21/0x40 >>> I [<c012fed0>] lock_timer_base+0x20/0x50 >>> I [<c0151563>] handle_IRQ_event+0x33/0xa0 >>> I [<c0151678>] __do_IRQ+0xa8/0x120 >>> I [<c01076e1>] do_IRQ+0x31/0x80 >>> I [<c02af6b0>] neigh_periodic_timer+0x0/0x140 >>> I [<c024ced5>] evtchn_do_upcall+0xe5/0x1f0 >>> I [<c0115fb0>] do_fixup_4gb_segment+0x0/0x170 >>> I [<c0105ba3>] hypervisor_callback+0x33/0x3b >>> I [<c0115fb0>] do_fixup_4gb_segment+0x0/0x170 >>> I [<c01ed68b>] delay_tsc+0xb/0x20 >>> I [<c01ed6d6>] __delay+0x6/0x10 >>> I [<c01160f0>] do_fixup_4gb_segment+0x140/0x170 >>> I [<c0169597>] do_munmap+0x197/0x200 >>> I [<c01ee0a4>] copy_to_user+0x34/0x70 >>> I [<c0105b6b>] error_code+0x2b/0x30 >>> Continuing... >>> >>> >>> FYI, when I reverted changeset 20122 of xen-unstable, the guest OS >>> is booted up as before. >>> >>> Best regards, >>> I Kan >>> >>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >> >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> > >-------------------------------text/plain------------------------------- >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi George, I tested xen-unstable (changeset: 20128) and linux-2.6.18-xen (changeset: 931) with the following cases. debit-accounted-only.diff doesn''t fix the problem. I will send a trace data file to only you because the size of the file is big. Case1 : With debit-accounted-only.diff, cpu_weight dom0:domU = 256:256 | vcpus | cpu_cap | boot-up | | | | times[sec] | +-------+---------+------------+ | 1 | 0 | 57 | | 1 | 50 | 80 | | 1 | 100 | 56 | | 2 | 0 | 51 | | 2 | 50 | 93 | | 2 | 100 | 62 | | 2 | 150 | 54 | | 2 | 200 | 50 | Case2 : With debit-accounted-only.diff, cpu_weight dom0:domU = 256:512 | vcpus | cpu_cap | boot-up | | | | times[sec] | +-------+---------+------------+ | 1 | 0 | 57 | | 1 | 50 | 725 | Slow!! | 1 | 100 | 57 | | 2 | 0 | 51 | | 2 | 50 | 1,000 over | I gave up the measurement. | 2 | 100 | 784 | Slow!! | 2 | 150 | 567 | Slow!! | 2 | 200 | 51 | Case3 : No patch, cpu_weight dom0:domU = 256:256 | vcpus | cpu_cap | boot-up | | | | times[sec] | +-------+---------+------------+ | 1 | 0 | 57 | | 1 | 50 | 80 | | 1 | 100 | 57 | | 2 | 0 | 50 | | 2 | 50 | 95 | | 2 | 100 | 61 | | 2 | 150 | 53 | | 2 | 200 | 50 | Case4 : No patch, cpu_weight dom0:domU = 256:512 | vcpus | cpu_cap | boot-up | | | | times[sec] | +-------+---------+------------+ | 1 | 0 | 57 | | 1 | 50 | 575 | Slow!! | 1 | 100 | 57 | | 2 | 0 | 50 | | 2 | 50 | 594 | Slow!! | 2 | 100 | 450 | Slow!! | 2 | 150 | 290 | Slow!! | 2 | 200 | 51 | Case5 : Without changeset 20122, cpu_weight dom0:domU = 256:256 | vcpus | cpu_cap | boot-up | | | | times[sec] | +-------+---------+------------+ | 1 | 0 | 57 | | 1 | 50 | 80 | | 1 | 100 | 56 | | 2 | 0 | 50 | | 2 | 50 | 95 | | 2 | 100 | 61 | | 2 | 150 | 53 | | 2 | 200 | 50 | Case6 : Without changeset 20122, cpu_weight dom0:domU = 256:512 | vcpus | cpu_cap | boot-up | | | | times[sec] | +-------+---------+------------+ | 1 | 0 | 56 | | 1 | 50 | 80 | | 1 | 100 | 56 | | 2 | 0 | 50 | | 2 | 50 | 95 | | 2 | 100 | 61 | | 2 | 150 | 53 | | 2 | 200 | 50 | Best regards, Kan Tue, 01 Sep 2009 08:41:39 +0900, Masaki Kanno wrote:>Hi George, > >I will try them on today. > > Kan > >Mon, 31 Aug 2009 14:10:47 +0100, George Dunlap wrote: > >>Hmm, it appears my patch still debits credits for VMs that aren''t >>earning credits anymore; such VMs can earn an unlimited amount of >>negative credit before becoming active again. >> >>Try the attached patches; if it doesn''t fix the problem, please take a >>short trace during boot using the following command and send it to me: >> >># xentrace -D -e 0x2f000 -S 128 -s 1000 /tmp/sched-boot.trace >> >>Thanks, >> -George >> >>On Mon, Aug 31, 2009 at 9:34 AM, Keir Fraser<keir.fraser@eu.citrix.com> >>wrote: >>> Hi Maskai, >>> >>> Thanks for the report. It''s a good idea to Cc the patch author when making >>> bug reports. In this case it''s George Dunlap who I''ve cc''ed in this reply. >>> >>> I -- Keir >>> >>> On 31/08/2009 07:50, "Masaki Kanno" <kanno.masaki@jp.fujitsu.com> wrote: >>> >>>> Hi, >>>> >>>> I tested the latest xen-unstable (changeset: 20128) and the latest >>>> linux-2.6.18-xen (changeset: 931). I A guest OS is booted up slower >>>> than before. I And the following messages are shown by the guest OS. >>>> >>>> Switching to new root and running init. >>>> unmounting old /dev >>>> unmounting old /proc >>>> unmounting old /sys >>>> >>>> I *************************************************************** >>>> I *************************************************************** >>>> I ** WARNING: Currently emulating unsupported memory accesses I ** >>>> I ** I I I I I in /lib/tls glibc libraries. The emulation is I I ** >>>> I ** I I I I I slow. To ensure full performance you should I I I ** >>>> I ** I I I I I install a ''xen-friendly'' (nosegneg) version of I ** >>>> I ** I I I I I the library, or disable tls support by executing ** >>>> I ** I I I I I the following as root: I I I I I I I I I I I I I ** >>>> I ** I I I I I mv /lib/tls /lib/tls.disabled I I I I I I I I I I ** >>>> I ** Offending process: modprobe (pid=761) I I I I I I I I I I ** >>>> I *************************************************************** >>>> I *************************************************************** >>>> >>>> Pausing... 5<3>BUG: soft lockup detected on CPU#0! >>>> I [<c0151205>] softlockup_tick+0xa5/0xd0 >>>> I [<c010978a>] timer_interrupt+0x2fa/0x6c0 >>>> I [<c011d8c1>] __activate_task+0x21/0x40 >>>> I [<c012fed0>] lock_timer_base+0x20/0x50 >>>> I [<c0151563>] handle_IRQ_event+0x33/0xa0 >>>> I [<c0151678>] __do_IRQ+0xa8/0x120 >>>> I [<c01076e1>] do_IRQ+0x31/0x80 >>>> I [<c02af6b0>] neigh_periodic_timer+0x0/0x140 >>>> I [<c024ced5>] evtchn_do_upcall+0xe5/0x1f0 >>>> I [<c0115fb0>] do_fixup_4gb_segment+0x0/0x170 >>>> I [<c0105ba3>] hypervisor_callback+0x33/0x3b >>>> I [<c0115fb0>] do_fixup_4gb_segment+0x0/0x170 >>>> I [<c01ed68b>] delay_tsc+0xb/0x20 >>>> I [<c01ed6d6>] __delay+0x6/0x10 >>>> I [<c01160f0>] do_fixup_4gb_segment+0x140/0x170 >>>> I [<c0169597>] do_munmap+0x197/0x200 >>>> I [<c01ee0a4>] copy_to_user+0x34/0x70 >>>> I [<c0105b6b>] error_code+0x2b/0x30 >>>> Continuing... >>>> >>>> >>>> FYI, when I reverted changeset 20122 of xen-unstable, the guest OS >>>> is booted up as before. >>>> >>>> Best regards, >>>> I Kan >>>> >>>> >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xensource.com >>>> http://lists.xensource.com/xen-devel >>> >>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >>> >> >>-------------------------------text/plain------------------------------- >>_______________________________________________ >>Xen-devel mailing list >>Xen-devel@lists.xensource.com >>http://lists.xensource.com/xen-devel > > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks Masaki! I've taken a quick look at the trace and haven't seen anything really obvious. Unfortunately I have some urgent work to do for XenServer, so this will have to take a back burner for a week or so. If anyone else is willing to investigate the problem, I certainly won't object. :-) -George 2009/9/1 Masaki Kanno <kanno.masaki@jp.fujitsu.com>:> Hi George, > > I tested xen-unstable (changeset: 20128) and linux-2.6.18-xen > (changeset: 931) with the following cases. > debit-accounted-only.diff doesn't fix the problem. > I will send a trace data file to only you because the size of > the file is big. > > Case1 : With debit-accounted-only.diff, cpu_weight dom0:domU = 256:256 > | vcpus | cpu_cap | boot-up | > | | | times[sec] | > +-------+---------+------------+ > | 1 | 0 | 57 | > | 1 | 50 | 80 | > | 1 | 100 | 56 | > | 2 | 0 | 51 | > | 2 | 50 | 93 | > | 2 | 100 | 62 | > | 2 | 150 | 54 | > | 2 | 200 | 50 | > > Case2 : With debit-accounted-only.diff, cpu_weight dom0:domU = 256:512 > | vcpus | cpu_cap | boot-up | > | | | times[sec] | > +-------+---------+------------+ > | 1 | 0 | 57 | > | 1 | 50 | 725 | Slow!! > | 1 | 100 | 57 | > | 2 | 0 | 51 | > | 2 | 50 | 1,000 over | I gave up the measurement. > | 2 | 100 | 784 | Slow!! > | 2 | 150 | 567 | Slow!! > | 2 | 200 | 51 | > > Case3 : No patch, cpu_weight dom0:domU = 256:256 > | vcpus | cpu_cap | boot-up | > | | | times[sec] | > +-------+---------+------------+ > | 1 | 0 | 57 | > | 1 | 50 | 80 | > | 1 | 100 | 57 | > | 2 | 0 | 50 | > | 2 | 50 | 95 | > | 2 | 100 | 61 | > | 2 | 150 | 53 | > | 2 | 200 | 50 | > > Case4 : No patch, cpu_weight dom0:domU = 256:512 > | vcpus | cpu_cap | boot-up | > | | | times[sec] | > +-------+---------+------------+ > | 1 | 0 | 57 | > | 1 | 50 | 575 | Slow!! > | 1 | 100 | 57 | > | 2 | 0 | 50 | > | 2 | 50 | 594 | Slow!! > | 2 | 100 | 450 | Slow!! > | 2 | 150 | 290 | Slow!! > | 2 | 200 | 51 | > > Case5 : Without changeset 20122, cpu_weight dom0:domU = 256:256 > | vcpus | cpu_cap | boot-up | > | | | times[sec] | > +-------+---------+------------+ > | 1 | 0 | 57 | > | 1 | 50 | 80 | > | 1 | 100 | 56 | > | 2 | 0 | 50 | > | 2 | 50 | 95 | > | 2 | 100 | 61 | > | 2 | 150 | 53 | > | 2 | 200 | 50 | > > Case6 : Without changeset 20122, cpu_weight dom0:domU = 256:512 > | vcpus | cpu_cap | boot-up | > | | | times[sec] | > +-------+---------+------------+ > | 1 | 0 | 56 | > | 1 | 50 | 80 | > | 1 | 100 | 56 | > | 2 | 0 | 50 | > | 2 | 50 | 95 | > | 2 | 100 | 61 | > | 2 | 150 | 53 | > | 2 | 200 | 50 | > > > Best regards, > Kan > > Tue, 01 Sep 2009 08:41:39 +0900, Masaki Kanno wrote: > >>Hi George, >> >>I will try them on today. >> >> Kan >> >>Mon, 31 Aug 2009 14:10:47 +0100, George Dunlap wrote: >> >>>Hmm, it appears my patch still debits credits for VMs that aren't >>>earning credits anymore; such VMs can earn an unlimited amount of >>>negative credit before becoming active again. >>> >>>Try the attached patches; if it doesn't fix the problem, please take a >>>short trace during boot using the following command and send it to me: >>> >>># xentrace -D -e 0x2f000 -S 128 -s 1000 /tmp/sched-boot.trace >>> >>>Thanks, >>> -George >>> >>>On Mon, Aug 31, 2009 at 9:34 AM, Keir Fraser<keir.fraser@eu.citrix.com> >>>wrote: >>>> Hi Maskai, >>>> >>>> Thanks for the report. It's a good idea to Cc the patch author when making >>>> bug reports. In this case it's George Dunlap who I've cc'ed in this reply. >>>> >>>> �-- Keir >>>> >>>> On 31/08/2009 07:50, "Masaki Kanno" <kanno.masaki@jp.fujitsu.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I tested the latest xen-unstable (changeset: 20128) and the latest >>>>> linux-2.6.18-xen (changeset: 931). �A guest OS is booted up slower >>>>> than before. �And the following messages are shown by the guest OS. >>>>> >>>>> Switching to new root and running init. >>>>> unmounting old /dev >>>>> unmounting old /proc >>>>> unmounting old /sys >>>>> >>>>> � *************************************************************** >>>>> � *************************************************************** >>>>> � ** WARNING: Currently emulating unsupported memory accesses �** >>>>> � ** � � � � �in /lib/tls glibc libraries. The emulation is � �** >>>>> � ** � � � � �slow. To ensure full performance you should � � �** >>>>> � ** � � � � �install a 'xen-friendly' (nosegneg) version of � ** >>>>> � ** � � � � �the library, or disable tls support by executing ** >>>>> � ** � � � � �the following as root: � � � � � � � � � � � � � ** >>>>> � ** � � � � �mv /lib/tls /lib/tls.disabled � � � � � � � � � �** >>>>> � ** Offending process: modprobe (pid=761) � � � � � � � � � � ** >>>>> � *************************************************************** >>>>> � *************************************************************** >>>>> >>>>> Pausing... 5<3>BUG: soft lockup detected on CPU#0! >>>>> �[<c0151205>] softlockup_tick+0xa5/0xd0 >>>>> �[<c010978a>] timer_interrupt+0x2fa/0x6c0 >>>>> �[<c011d8c1>] __activate_task+0x21/0x40 >>>>> �[<c012fed0>] lock_timer_base+0x20/0x50 >>>>> �[<c0151563>] handle_IRQ_event+0x33/0xa0 >>>>> �[<c0151678>] __do_IRQ+0xa8/0x120 >>>>> �[<c01076e1>] do_IRQ+0x31/0x80 >>>>> �[<c02af6b0>] neigh_periodic_timer+0x0/0x140 >>>>> �[<c024ced5>] evtchn_do_upcall+0xe5/0x1f0 >>>>> �[<c0115fb0>] do_fixup_4gb_segment+0x0/0x170 >>>>> �[<c0105ba3>] hypervisor_callback+0x33/0x3b >>>>> �[<c0115fb0>] do_fixup_4gb_segment+0x0/0x170 >>>>> �[<c01ed68b>] delay_tsc+0xb/0x20 >>>>> �[<c01ed6d6>] __delay+0x6/0x10 >>>>> �[<c01160f0>] do_fixup_4gb_segment+0x140/0x170 >>>>> �[<c0169597>] do_munmap+0x197/0x200 >>>>> �[<c01ee0a4>] copy_to_user+0x34/0x70 >>>>> �[<c0105b6b>] error_code+0x2b/0x30 >>>>> Continuing... >>>>> >>>>> >>>>> FYI, when I reverted changeset 20122 of xen-unstable, the guest OS >>>>> is booted up as before. >>>>> >>>>> Best regards, >>>>> �Kan >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Xen-devel mailing list >>>>> Xen-devel@lists.xensource.com >>>>> http://lists.xensource.com/xen-devel >>>> >>>> >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xensource.com >>>> http://lists.xensource.com/xen-devel >>>> >>> >>>-------------------------------text/plain------------------------------- >>>_______________________________________________ >>>Xen-devel mailing list >>>Xen-devel@lists.xensource.com >>>http://lists.xensource.com/xen-devel >> >> >>_______________________________________________ >>Xen-devel mailing list >>Xen-devel@lists.xensource.com >>http://lists.xensource.com/xen-devel > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi George, Keir, I confirmed the problem was fixed by changeset 20270. Thanks, Keir!! Best regards, Kan Wed, 2 Sep 2009 11:48:22 +0100, George Dunlap wrote:>Thanks Masaki! I''ve taken a quick look at the trace and haven''t seen >anything really obvious. Unfortunately I have some urgent work to do >for XenServer, so this will have to take a back burner for a week or >so. > >If anyone else is willing to investigate the problem, I certainly >won''t object. :-) > > -George > >2009/9/1 Masaki Kanno <kanno.masaki@jp.fujitsu.com>: >> Hi George, >> >> I tested xen-unstable (changeset: 20128) and linux-2.6.18-xen >> (changeset: 931) with the following cases. >> debit-accounted-only.diff doesn''t fix the problem. >> I will send a trace data file to only you because the size of >> the file is big. >> >> Case1 : With debit-accounted-only.diff, cpu_weight dom0:domU = 256:256 >> ? ? ?| vcpus | cpu_cap | boot-up ? ?| >> ? ? ?| ? ? ? | ? ? ? ? | times[sec] | >> ? ? ?+-------+---------+------------+ >> ? ? ?| ? 1 ? | ? ? 0 ? | ? ? 57 ? ? | >> ? ? ?| ? 1 ? | ? ?50 ? | ? ? 80 ? ? | >> ? ? ?| ? 1 ? | ? 100 ? | ? ? 56 ? ? | >> ? ? ?| ? 2 ? | ? ? 0 ? | ? ? 51 ? ? | >> ? ? ?| ? 2 ? | ? ?50 ? | ? ? 93 ? ? | >> ? ? ?| ? 2 ? | ? 100 ? | ? ? 62 ? ? | >> ? ? ?| ? 2 ? | ? 150 ? | ? ? 54 ? ? | >> ? ? ?| ? 2 ? | ? 200 ? | ? ? 50 ? ? | >> >> Case2 : With debit-accounted-only.diff, cpu_weight dom0:domU = 256:512 >> ? ? ?| vcpus | cpu_cap | boot-up ? ?| >> ? ? ?| ? ? ? | ? ? ? ? | times[sec] | >> ? ? ?+-------+---------+------------+ >> ? ? ?| ? 1 ? | ? ? 0 ? | ? ? 57 ? ? | >> ? ? ?| ? 1 ? | ? ?50 ? | ? ?725 ? ? | Slow!! >> ? ? ?| ? 1 ? | ? 100 ? | ? ? 57 ? ? | >> ? ? ?| ? 2 ? | ? ? 0 ? | ? ? 51 ? ? | >> ? ? ?| ? 2 ? | ? ?50 ? | 1,000 over | I gave up the measurement. >> ? ? ?| ? 2 ? | ? 100 ? | ? ?784 ? ? | Slow!! >> ? ? ?| ? 2 ? | ? 150 ? | ? ?567 ? ? | Slow!! >> ? ? ?| ? 2 ? | ? 200 ? | ? ? 51 ? ? | >> >> Case3 : No patch, cpu_weight dom0:domU = 256:256 >> ? ? ?| vcpus | cpu_cap | boot-up ? ?| >> ? ? ?| ? ? ? | ? ? ? ? | times[sec] | >> ? ? ?+-------+---------+------------+ >> ? ? ?| ? 1 ? | ? ? 0 ? | ? ? 57 ? ? | >> ? ? ?| ? 1 ? | ? ?50 ? | ? ? 80 ? ? | >> ? ? ?| ? 1 ? | ? 100 ? | ? ? 57 ? ? | >> ? ? ?| ? 2 ? | ? ? 0 ? | ? ? 50 ? ? | >> ? ? ?| ? 2 ? | ? ?50 ? | ? ? 95 ? ? | >> ? ? ?| ? 2 ? | ? 100 ? | ? ? 61 ? ? | >> ? ? ?| ? 2 ? | ? 150 ? | ? ? 53 ? ? | >> ? ? ?| ? 2 ? | ? 200 ? | ? ? 50 ? ? | >> >> Case4 : No patch, cpu_weight dom0:domU = 256:512 >> ? ? ?| vcpus | cpu_cap | boot-up ? ?| >> ? ? ?| ? ? ? | ? ? ? ? | times[sec] | >> ? ? ?+-------+---------+------------+ >> ? ? ?| ? 1 ? | ? ? 0 ? | ? ? 57 ? ? | >> ? ? ?| ? 1 ? | ? ?50 ? | ? ?575 ? ? | Slow!! >> ? ? ?| ? 1 ? | ? 100 ? | ? ? 57 ? ? | >> ? ? ?| ? 2 ? | ? ? 0 ? | ? ? 50 ? ? | >> ? ? ?| ? 2 ? | ? ?50 ? | ? ?594 ? ? | Slow!! >> ? ? ?| ? 2 ? | ? 100 ? | ? ?450 ? ? | Slow!! >> ? ? ?| ? 2 ? | ? 150 ? | ? ?290 ? ? | Slow!! >> ? ? ?| ? 2 ? | ? 200 ? | ? ? 51 ? ? | >> >> Case5 : Without changeset 20122, cpu_weight dom0:domU = 256:256 >> ? ? ?| vcpus | cpu_cap | boot-up ? ?| >> ? ? ?| ? ? ? | ? ? ? ? | times[sec] | >> ? ? ?+-------+---------+------------+ >> ? ? ?| ? 1 ? | ? ? 0 ? | ? ? 57 ? ? | >> ? ? ?| ? 1 ? | ? ?50 ? | ? ? 80 ? ? | >> ? ? ?| ? 1 ? | ? 100 ? | ? ? 56 ? ? | >> ? ? ?| ? 2 ? | ? ? 0 ? | ? ? 50 ? ? | >> ? ? ?| ? 2 ? | ? ?50 ? | ? ? 95 ? ? | >> ? ? ?| ? 2 ? | ? 100 ? | ? ? 61 ? ? | >> ? ? ?| ? 2 ? | ? 150 ? | ? ? 53 ? ? | >> ? ? ?| ? 2 ? | ? 200 ? | ? ? 50 ? ? | >> >> Case6 : Without changeset 20122, cpu_weight dom0:domU = 256:512 >> ? ? ?| vcpus | cpu_cap | boot-up ? ?| >> ? ? ?| ? ? ? | ? ? ? ? | times[sec] | >> ? ? ?+-------+---------+------------+ >> ? ? ?| ? 1 ? | ? ? 0 ? | ? ? 56 ? ? | >> ? ? ?| ? 1 ? | ? ?50 ? | ? ? 80 ? ? | >> ? ? ?| ? 1 ? | ? 100 ? | ? ? 56 ? ? | >> ? ? ?| ? 2 ? | ? ? 0 ? | ? ? 50 ? ? | >> ? ? ?| ? 2 ? | ? ?50 ? | ? ? 95 ? ? | >> ? ? ?| ? 2 ? | ? 100 ? | ? ? 61 ? ? | >> ? ? ?| ? 2 ? | ? 150 ? | ? ? 53 ? ? | >> ? ? ?| ? 2 ? | ? 200 ? | ? ? 50 ? ? | >> >> >> Best regards, >> ?Kan >> >> Tue, 01 Sep 2009 08:41:39 +0900, Masaki Kanno wrote: >> >>>Hi George, >>> >>>I will try them on today. >>> >>> Kan >>> >>>Mon, 31 Aug 2009 14:10:47 +0100, George Dunlap wrote: >>> >>>>Hmm, it appears my patch still debits credits for VMs that aren''t >>>>earning credits anymore; such VMs can earn an unlimited amount of >>>>negative credit before becoming active again. >>>> >>>>Try the attached patches; if it doesn''t fix the problem, please take a >>>>short trace during boot using the following command and send it to me: >>>> >>>># xentrace -D -e 0x2f000 -S 128 -s 1000 /tmp/sched-boot.trace >>>> >>>>Thanks, >>>> -George >>>> >>>>On Mon, Aug 31, 2009 at 9:34 AM, Keir Fraser<keir.fraser@eu.citrix.com> >>>>wrote: >>>>> Hi Maskai, >>>>> >>>>> Thanks for the report. It''s a good idea to Cc the patch author when >>>>> making >>>>> bug reports. In this case it''s George Dunlap who I''ve cc''ed in this >>>>> reply. >>>>> >>>>> ?-- Keir >>>>> >>>>> On 31/08/2009 07:50, "Masaki Kanno" <kanno.masaki@jp.fujitsu.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I tested the latest xen-unstable (changeset: 20128) and the latest >>>>>> linux-2.6.18-xen (changeset: 931). ?A guest OS is booted up slower >>>>>> than before. ?And the following messages are shown by the guest OS. >>>>>> >>>>>> Switching to new root and running init. >>>>>> unmounting old /dev >>>>>> unmounting old /proc >>>>>> unmounting old /sys >>>>>> >>>>>> ? *************************************************************** >>>>>> ? *************************************************************** >>>>>> ? ** WARNING: Currently emulating unsupported memory accesses ?** >>>>>> ? ** ? ? ? ? ?in /lib/tls glibc libraries. The emulation is ? ?** >>>>>> ? ** ? ? ? ? ?slow. To ensure full performance you should ? ? ?** >>>>>> ? ** ? ? ? ? ?install a ''xen-friendly'' (nosegneg) version of ? ** >>>>>> ? ** ? ? ? ? ?the library, or disable tls support by executing ** >>>>>> ? ** ? ? ? ? ?the following as root: ? ? ? ? ? ? ? ? ? ? ? ? ? ** >>>>>> ? ** ? ? ? ? ?mv /lib/tls /lib/tls.disabled ? ? ? ? ? ? ? ? ? ?** >>>>>> ? ** Offending process: modprobe (pid=761) ? ? ? ? ? ? ? ? ? ? ** >>>>>> ? *************************************************************** >>>>>> ? *************************************************************** >>>>>> >>>>>> Pausing... 5<3>BUG: soft lockup detected on CPU#0! >>>>>> ?[<c0151205>] softlockup_tick+0xa5/0xd0 >>>>>> ?[<c010978a>] timer_interrupt+0x2fa/0x6c0 >>>>>> ?[<c011d8c1>] __activate_task+0x21/0x40 >>>>>> ?[<c012fed0>] lock_timer_base+0x20/0x50 >>>>>> ?[<c0151563>] handle_IRQ_event+0x33/0xa0 >>>>>> ?[<c0151678>] __do_IRQ+0xa8/0x120 >>>>>> ?[<c01076e1>] do_IRQ+0x31/0x80 >>>>>> ?[<c02af6b0>] neigh_periodic_timer+0x0/0x140 >>>>>> ?[<c024ced5>] evtchn_do_upcall+0xe5/0x1f0 >>>>>> ?[<c0115fb0>] do_fixup_4gb_segment+0x0/0x170 >>>>>> ?[<c0105ba3>] hypervisor_callback+0x33/0x3b >>>>>> ?[<c0115fb0>] do_fixup_4gb_segment+0x0/0x170 >>>>>> ?[<c01ed68b>] delay_tsc+0xb/0x20 >>>>>> ?[<c01ed6d6>] __delay+0x6/0x10 >>>>>> ?[<c01160f0>] do_fixup_4gb_segment+0x140/0x170 >>>>>> ?[<c0169597>] do_munmap+0x197/0x200 >>>>>> ?[<c01ee0a4>] copy_to_user+0x34/0x70 >>>>>> ?[<c0105b6b>] error_code+0x2b/0x30 >>>>>> Continuing... >>>>>> >>>>>> >>>>>> FYI, when I reverted changeset 20122 of xen-unstable, the guest OS >>>>>> is booted up as before. >>>>>> >>>>>> Best regards, >>>>>> ?Kan >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Xen-devel mailing list >>>>>> Xen-devel@lists.xensource.com >>>>>> http://lists.xensource.com/xen-devel >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Xen-devel mailing list >>>>> Xen-devel@lists.xensource.com >>>>> http://lists.xensource.com/xen-devel >>>>> >>>> >>>>-------------------------------text/plain------------------------------- >>>>_______________________________________________ >>>>Xen-devel mailing list >>>>Xen-devel@lists.xensource.com >>>>http://lists.xensource.com/xen-devel >>> >>> >>>_______________________________________________ >>>Xen-devel mailing list >>>Xen-devel@lists.xensource.com >>>http://lists.xensource.com/xen-devel >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel