Peter Maloney
2012-Aug-06 10:12 UTC
4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow
my AMD FX-8150 system with vanilla source code is super slow, both the dom0 and domUs. However, after I merge the upstream patches I found in the openSUSE rpm, it runs normally. I tried 4.2-unstable and it was the same. There was no rc1 when I tested it about 1.5 weeks ago. And 4.2 has the same horrible performance, and obviously those patches won''t work any more since the 4.2 code looks completely reorganized, so I''m stuck with 4.1.2 Here is the rpm I was using at the time: http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm To see the list of the patches and what order to apply them, see the spec file. Please make sure this performance issue is fixed for the 4.2 release. And I would be happy to test whatever files you send me.
Andrew Cooper
2012-Aug-06 10:24 UTC
Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow
On 06/08/12 11:12, Peter Maloney wrote:> my AMD FX-8150 system with vanilla source code is super slow, both the > dom0 and domUs. However, after I merge the upstream patches I found in > the openSUSE rpm, it runs normally. > > I tried 4.2-unstable and it was the same. There was no rc1 when I tested > it about 1.5 weeks ago. And 4.2 has the same horrible performance, and > obviously those patches won''t work any more since the 4.2 code looks > completely reorganized, so I''m stuck with 4.1.2 > > Here is the rpm I was using at the time: > http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm > > To see the list of the patches and what order to apply them, see the > spec file. > > Please make sure this performance issue is fixed for the 4.2 release. > And I would be happy to test whatever files you send me. >Without identifying which patch or patches make a difference for you, there is very little we can do. There are 406 patches in that spec file. Furthermore, from the file names, I would say that most of the patches have been backported from unstable.> > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel-- Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer T: +44 (0)1223 225 900, http://www.citrix.com
Jan Beulich
2012-Aug-06 10:31 UTC
Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow
>>> On 06.08.12 at 12:12, Peter Maloney <peter.maloney@brockmann-consult.de> wrote: > my AMD FX-8150 system with vanilla source code is super slow, both the > dom0 and domUs. However, after I merge the upstream patches I found in > the openSUSE rpm, it runs normally.I''d be very surprised if you really just took the upstream patches, and the result was better than 4.2-rc1. After all, what upstream means is that they were taken from -unstable.> I tried 4.2-unstable and it was the same. There was no rc1 when I tested > it about 1.5 weeks ago. And 4.2 has the same horrible performance, and > obviously those patches won''t work any more since the 4.2 code looks > completely reorganized, so I''m stuck with 4.1.2Obviously the upstream patches can''t be applied to something that already has all those changes. Other patches, of which we unfortunately have quite a few, would be a different story.> Here is the rpm I was using at the time: > http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm > > To see the list of the patches and what order to apply them, see the > spec file.That still won''t tell us which patches you did apply.> Please make sure this performance issue is fixed for the 4.2 release. > And I would be happy to test whatever files you send me.The sort of report you''re doing isn''t that helpful. What would help is if you could narrow down which patch(es) it is that make things so much better. Giving 4.1.3-rc a try might also be worthwhile, albeit I would hope we don''t have a regression in 4.2.0-rc compared to 4.1.3-rc... Jan
Malcolm Crossley
2012-Aug-06 12:41 UTC
Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow
On 06/08/12 11:12, Peter Maloney wrote:> my AMD FX-8150 system with vanilla source code is super slow, both the > dom0 and domUs. However, after I merge the upstream patches I found in > the openSUSE rpm, it runs normally. > > I tried 4.2-unstable and it was the same. There was no rc1 when I tested > it about 1.5 weeks ago. And 4.2 has the same horrible performance, and > obviously those patches won''t work any more since the 4.2 code looks > completely reorganized, so I''m stuck with 4.1.2 > > Here is the rpm I was using at the time: > http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm > > To see the list of the patches and what order to apply them, see the > spec file. > > Please make sure this performance issue is fixed for the 4.2 release. > And I would be happy to test whatever files you send me. > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-develI suspect you may need the following patch to improve your 4.1.2 performance: http://xenbits.xen.org/hg/xen-4.1-testing.hg/rev/435493696053 The cache flush on every C2 transition is very expensive and causes a large slow down. 4.1.3-rc3 already includes that patch so it would be worth testing that version. Malcolm
Konrad Rzeszutek Wilk
2012-Aug-06 14:02 UTC
Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow
On Mon, Aug 06, 2012 at 01:41:07PM +0100, Malcolm Crossley wrote:> On 06/08/12 11:12, Peter Maloney wrote: > >my AMD FX-8150 system with vanilla source code is super slow, both the > >dom0 and domUs. However, after I merge the upstream patches I found in > >the openSUSE rpm, it runs normally. > > > >I tried 4.2-unstable and it was the same. There was no rc1 when I tested > >it about 1.5 weeks ago. And 4.2 has the same horrible performance, and > >obviously those patches won''t work any more since the 4.2 code looks > >completely reorganized, so I''m stuck with 4.1.2 > > > >Here is the rpm I was using at the time: > >http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm > > > >To see the list of the patches and what order to apply them, see the > >spec file. > > > >Please make sure this performance issue is fixed for the 4.2 release. > >And I would be happy to test whatever files you send me. > > > > > > > >_______________________________________________ > >Xen-devel mailing list > >Xen-devel@lists.xen.org > >http://lists.xen.org/xen-devel > I suspect you may need the following patch to improve your 4.1.2 > performance: > > http://xenbits.xen.org/hg/xen-4.1-testing.hg/rev/435493696053 > > The cache flush on every C2 transition is very expensive and causes > a large slow down. > > 4.1.3-rc3 already includes that patch so it would be worth testing > that version.MA Young, could this be back-ported in the F17 and F16. I belive Micahel Petullo setup a bug for that?> > Malcolm > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Peter Maloney
2012-Aug-07 07:25 UTC
Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow
> That still won''t tell us which patches you did apply.I applied no patches and tested, and the result was slow. And then applied all patches, and it was fast. I didn''t try figuring out which one it was. So I guess I''ll try: - the latest unstable 4.2 - the 4.1.3-rc (Which includes the patch Malcolm suggested) - and my rpm source with half patches, 3/4 of them, etc. binary search style to see which patch(es) changed the performance. But this means I won''t be able to narrow it down to a single patch, but only the point in the long list where the most dramatic change happens, possibly depending on many previous patches. Thanks so far, guys. On 08/06/2012 12:31 PM, Jan Beulich wrote:>>>> On 06.08.12 at 12:12, Peter Maloney <peter.maloney@brockmann-consult.de> wrote: >> my AMD FX-8150 system with vanilla source code is super slow, both the >> dom0 and domUs. However, after I merge the upstream patches I found in >> the openSUSE rpm, it runs normally. > I''d be very surprised if you really just took the upstream patches, > and the result was better than 4.2-rc1. After all, what upstream > means is that they were taken from -unstable. > >> I tried 4.2-unstable and it was the same. There was no rc1 when I tested >> it about 1.5 weeks ago. And 4.2 has the same horrible performance, and >> obviously those patches won''t work any more since the 4.2 code looks >> completely reorganized, so I''m stuck with 4.1.2 > Obviously the upstream patches can''t be applied to something > that already has all those changes. Other patches, of which we > unfortunately have quite a few, would be a different story. > >> Here is the rpm I was using at the time: >> http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm >> >> To see the list of the patches and what order to apply them, see the >> spec file. > That still won''t tell us which patches you did apply. > >> Please make sure this performance issue is fixed for the 4.2 release. >> And I would be happy to test whatever files you send me. > The sort of report you''re doing isn''t that helpful. What would > help is if you could narrow down which patch(es) it is that > make things so much better. Giving 4.1.3-rc a try might also > be worthwhile, albeit I would hope we don''t have a regression > in 4.2.0-rc compared to 4.1.3-rc... > > Jan > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel-- -------------------------------------------- Peter Maloney Brockmann Consult Max-Planck-Str. 2 21502 Geesthacht Germany Tel: +49 4152 889 300 Fax: +49 4152 889 333 E-mail: peter.maloney@brockmann-consult.de Internet: http://www.brockmann-consult.de --------------------------------------------
M A Young
2012-Aug-07 21:42 UTC
Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow
On Mon, 6 Aug 2012, Konrad Rzeszutek Wilk wrote:> On Mon, Aug 06, 2012 at 01:41:07PM +0100, Malcolm Crossley wrote:>> I suspect you may need the following patch to improve your 4.1.2 >> performance: >> >> http://xenbits.xen.org/hg/xen-4.1-testing.hg/rev/435493696053 >> >> The cache flush on every C2 transition is very expensive and causes >> a large slow down. >> >> 4.1.3-rc3 already includes that patch so it would be worth testing >> that version. > > MA Young, could this be back-ported in the F17 and F16. I belive > Micahel Petullo setup a bug for that?I don''t recall I seeing a bug for this but the patch is in xen-4.1.2-25.fc18 and xen-4.1.2-25.fc17 (which is building now). Michael Young
W. Michael Petullo
2012-Aug-07 22:21 UTC
Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow
>>> I suspect you may need the following patch to improve your 4.1.2 >>> performance: >>> >>> http://xenbits.xen.org/hg/xen-4.1-testing.hg/rev/435493696053 >>> >>> The cache flush on every C2 transition is very expensive and causes >>> a large slow down.>>> 4.1.3-rc3 already includes that patch so it would be worth testing >>> that version.>> MA Young, could this be back-ported in the F17 and F16. I belive >> Micahel Petullo setup a bug for that?> I don''t recall I seeing a bug for this but the patch is in > xen-4.1.2-25.fc18 and xen-4.1.2-25.fc17 (which is building now).The performance-related bug I filed is at: https://bugzilla.redhat.com/show_bug.cgi?id=841330 -- Mike :wq
Peter Maloney
2012-Aug-13 18:54 UTC
Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow
So... did my 4.2-unstable test, using a fresh pull from yesterday; dom0 is normal fast (unlike previous tests), and domU is ultra slow, but actually boots, and graphics card passthrough works without any patches, and so does the USB keyboard, but USB mouse passthrough doesn''t work. On 08/07/2012 09:25 AM, Peter Maloney wrote:>> That still won''t tell us which patches you did apply. > I applied no patches and tested, and the result was slow. And then > applied all patches, and it was fast. I didn''t try figuring out which > one it was. > > > So I guess I''ll try: > - the latest unstable 4.2 > - the 4.1.3-rc (Which includes the patch Malcolm suggested) > - and my rpm source with half patches, 3/4 of them, etc. binary search > style to see which patch(es) changed the performance. But this means I > won''t be able to narrow it down to a single patch, but only the point in > the long list where the most dramatic change happens, possibly depending > on many previous patches. > > Thanks so far, guys. > > > On 08/06/2012 12:31 PM, Jan Beulich wrote: >>>>> On 06.08.12 at 12:12, Peter Maloney <peter.maloney@brockmann-consult.de> wrote: >>> my AMD FX-8150 system with vanilla source code is super slow, both the >>> dom0 and domUs. However, after I merge the upstream patches I found in >>> the openSUSE rpm, it runs normally. >> I''d be very surprised if you really just took the upstream patches, >> and the result was better than 4.2-rc1. After all, what upstream >> means is that they were taken from -unstable. >> >>> I tried 4.2-unstable and it was the same. There was no rc1 when I tested >>> it about 1.5 weeks ago. And 4.2 has the same horrible performance, and >>> obviously those patches won''t work any more since the 4.2 code looks >>> completely reorganized, so I''m stuck with 4.1.2 >> Obviously the upstream patches can''t be applied to something >> that already has all those changes. Other patches, of which we >> unfortunately have quite a few, would be a different story. >> >>> Here is the rpm I was using at the time: >>> http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm >>> >>> To see the list of the patches and what order to apply them, see the >>> spec file. >> That still won''t tell us which patches you did apply. >> >>> Please make sure this performance issue is fixed for the 4.2 release. >>> And I would be happy to test whatever files you send me. >> The sort of report you''re doing isn''t that helpful. What would >> help is if you could narrow down which patch(es) it is that >> make things so much better. Giving 4.1.3-rc a try might also >> be worthwhile, albeit I would hope we don''t have a regression >> in 4.2.0-rc compared to 4.1.3-rc... >> >> Jan >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel >
Peter Maloney
2012-Aug-13 20:59 UTC
Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow
I also tested 4.1.3, which is fast, and both USB and graphics passthrough work, but "xl create" gave this message the first time I started the vm (but not the second): libxl: error: libxl_pci.c:750:libxl_device_pci_reset The kernel doesn''t support reset from sysfs for PCI device 0000:00:12.0 0000:00:12.0 is a USB device, which works in the VM. peter:/opt # lspci -v | grep 00:12.0 00:12.0 USB Controller: ATI Technologies Inc SB7x0/SB8x0/SB9x0 USB OHCI0 Controller (prog-if 10 [OHCI]) On 08/13/2012 08:54 PM, Peter Maloney wrote:> So... did my 4.2-unstable test, using a fresh pull from yesterday; dom0 > is normal fast (unlike previous tests), and domU is ultra slow, but > actually boots, and graphics card passthrough works without any patches, > and so does the USB keyboard, but USB mouse passthrough doesn''t work. > > > On 08/07/2012 09:25 AM, Peter Maloney wrote: >>> That still won''t tell us which patches you did apply. >> I applied no patches and tested, and the result was slow. And then >> applied all patches, and it was fast. I didn''t try figuring out which >> one it was. >> >> >> So I guess I''ll try: >> - the latest unstable 4.2 >> - the 4.1.3-rc (Which includes the patch Malcolm suggested) >> - and my rpm source with half patches, 3/4 of them, etc. binary search >> style to see which patch(es) changed the performance. But this means I >> won''t be able to narrow it down to a single patch, but only the point in >> the long list where the most dramatic change happens, possibly depending >> on many previous patches. >> >> Thanks so far, guys. >> >> >> On 08/06/2012 12:31 PM, Jan Beulich wrote: >>>>>> On 06.08.12 at 12:12, Peter Maloney <peter.maloney@brockmann-consult.de> wrote: >>>> my AMD FX-8150 system with vanilla source code is super slow, both the >>>> dom0 and domUs. However, after I merge the upstream patches I found in >>>> the openSUSE rpm, it runs normally. >>> I''d be very surprised if you really just took the upstream patches, >>> and the result was better than 4.2-rc1. After all, what upstream >>> means is that they were taken from -unstable. >>> >>>> I tried 4.2-unstable and it was the same. There was no rc1 when I tested >>>> it about 1.5 weeks ago. And 4.2 has the same horrible performance, and >>>> obviously those patches won''t work any more since the 4.2 code looks >>>> completely reorganized, so I''m stuck with 4.1.2 >>> Obviously the upstream patches can''t be applied to something >>> that already has all those changes. Other patches, of which we >>> unfortunately have quite a few, would be a different story. >>> >>>> Here is the rpm I was using at the time: >>>> http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm >>>> >>>> To see the list of the patches and what order to apply them, see the >>>> spec file. >>> That still won''t tell us which patches you did apply. >>> >>>> Please make sure this performance issue is fixed for the 4.2 release. >>>> And I would be happy to test whatever files you send me. >>> The sort of report you''re doing isn''t that helpful. What would >>> help is if you could narrow down which patch(es) it is that >>> make things so much better. Giving 4.1.3-rc a try might also >>> be worthwhile, albeit I would hope we don''t have a regression >>> in 4.2.0-rc compared to 4.1.3-rc... >>> >>> Jan >>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xen.org >>> http://lists.xen.org/xen-devel > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Peter Maloney
2012-Oct-03 17:19 UTC
Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow
I ran some new tests... 4.1.2 with different patches, and
4.3-unstable.Some details are below.
At some point in the future, I will try some builds between 4.1 and 4.2
(but at the moment am not sure how with mercurial or what options I have).
4.1.2
short version: dom0 works fine; domu ran only in a few builds and works fine
long version:
I tested 4.1.2 again, with a few selections of patches (first n patches
where n was; 0, 23, 46, 93, 186, 279; there are 373 patches in total).
All of them ran fast in dom0, unlike when I first started this mailing
list thread, and the builds that would run my windows domu ran it fast.
So probably there was something strange with the kernel I had before,
which was probably 3.4.4; now I''m using something like 3.5.x
(cmason''s
linux-btrfs repository, for-linus branch)
4.3-unstable
short version: dom0 works fine; domu always runs terribly slow (which
leads me to wanting to test what changed between 4.1 and 4.2)
long version:
I pulled the latest source, built it, and dom0 is fast just like with
4.2, but windows hvm domu is still terribly slow, and (with 7 vcpus), it
consumes between 500-650% while booting and a few minutes afterwards.
With 4 CPUs, I would expect between 350-550% from observations with 4.2
but didn''t test other cpu counts with 4.3. (another side note, with
4.1.3 which is normally fast, it will run slow like 4.2 and 4.3 if I set
cpus="2,4,6,8" instead of cpus=4)
xentop - 11:32:44 Xen 4.3-unstable
2 domains: 2 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
Mem: 16757972k total, 16739788k used, 18184k free CPUs: 8 @ 4499MHz
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR
VBD_RSECT VBD_WSECT SSID
Domain-0 -----r 784 27.2 12314624 73.5 12582912
75.1 8 0 0 0 0 0 0
0 0 0 0
windowsxp2 -----r 3244 637.1 4197220 25.0 4198400
25.1 7 1 344 56 2 0 14381 6054
651283 122280 0
And then after idling for 10 minutes, it is under 200%
xentop - 11:35:29 Xen 4.3-unstable
2 domains: 1 running, 1 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
Mem: 16757972k total, 16739788k used, 18184k free CPUs: 8 @ 4499MHz
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR
VBD_RSECT VBD_WSECT SSID
Domain-0 -----r 839 42.4 12314624 73.5 12582912
75.1 8 0 0 0 0 0 0
0 0 0 0
windowsxp2 --b--- 3583 114.1 4197220 25.0 4198400
25.1 7 1 426 66 2 0 14408 7501
651853 180372 0
And then when it is in use (just loading a youtube page), it is up high
again.
xentop - 11:37:17 Xen 4.3-unstable
2 domains: 2 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
Mem: 16757972k total, 16739788k used, 18184k free CPUs: 8 @ 4499MHz
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR
VBD_RSECT VBD_WSECT SSID
Domain-0 -----r 875 37.8 12314624 73.5 12582912
75.1 8 0 0 0 0 0 0
0 0 0 0
windowsxp2 -----r 3945 529.7 4197220 25.0 4198400
25.1 7 1 4885 201 2 0 17096 8168
788573 198458 0
And also if I shut down the vm while it is at 600% cpu, it takes
something like 10-15 minutes to shut down.
and CPU temperature is only 45 degrees during the high cpu usage, and
26-32 degrees when it''s 0%, so whatever CPU waste it''s doing
is not
generating much heat. With 4.1.3, while a game is open, it reports 200%
cpu, and the temperature is around 50 degrees. I have a huge CPU cooler;
it''s overclocked; and normally it runs about 55-70 degrees using 8
cores
depending on the task.
I tested with apic=0 and apic=1 (apic=1 will run windows very slow in
4.1.x, so I have been using apic=0 normally)
On 08/13/2012 10:59 PM, Peter Maloney wrote:> I also tested 4.1.3, which is fast, and both USB and graphics
> passthrough work, but "xl create" gave this message the first
time I
> started the vm (but not the second):
>
> libxl: error: libxl_pci.c:750:libxl_device_pci_reset The kernel
doesn''t
> support reset from sysfs for PCI device 0000:00:12.0
>
>
> 0000:00:12.0 is a USB device, which works in the VM.
>
> peter:/opt # lspci -v | grep 00:12.0
> 00:12.0 USB Controller: ATI Technologies Inc SB7x0/SB8x0/SB9x0 USB OHCI0
> Controller (prog-if 10 [OHCI])
>
>
> On 08/13/2012 08:54 PM, Peter Maloney wrote:
>> So... did my 4.2-unstable test, using a fresh pull from yesterday; dom0
>> is normal fast (unlike previous tests), and domU is ultra slow, but
>> actually boots, and graphics card passthrough works without any
patches,
>> and so does the USB keyboard, but USB mouse passthrough
doesn''t work.
>>
>>
>> On 08/07/2012 09:25 AM, Peter Maloney wrote:
>>>> That still won''t tell us which patches you did apply.
>>> I applied no patches and tested, and the result was slow. And then
>>> applied all patches, and it was fast. I didn''t try
figuring out which
>>> one it was.
>>>
>>>
>>> So I guess I''ll try:
>>> - the latest unstable 4.2
>>> - the 4.1.3-rc (Which includes the patch Malcolm suggested)
>>> - and my rpm source with half patches, 3/4 of them, etc. binary
search
>>> style to see which patch(es) changed the performance. But this
means I
>>> won''t be able to narrow it down to a single patch, but
only the point in
>>> the long list where the most dramatic change happens, possibly
depending
>>> on many previous patches.
>>>
>>> Thanks so far, guys.
>>>
>>>
>>> On 08/06/2012 12:31 PM, Jan Beulich wrote:
>>>>>>> On 06.08.12 at 12:12, Peter Maloney
<peter.maloney@brockmann-consult.de> wrote:
>>>>> my AMD FX-8150 system with vanilla source code is super
slow, both the
>>>>> dom0 and domUs. However, after I merge the upstream patches
I found in
>>>>> the openSUSE rpm, it runs normally.
>>>> I''d be very surprised if you really just took the
upstream patches,
>>>> and the result was better than 4.2-rc1. After all, what
upstream
>>>> means is that they were taken from -unstable.
>>>>
>>>>> I tried 4.2-unstable and it was the same. There was no rc1
when I tested
>>>>> it about 1.5 weeks ago. And 4.2 has the same horrible
performance, and
>>>>> obviously those patches won''t work any more since
the 4.2 code looks
>>>>> completely reorganized, so I''m stuck with 4.1.2
>>>> Obviously the upstream patches can''t be applied to
something
>>>> that already has all those changes. Other patches, of which we
>>>> unfortunately have quite a few, would be a different story.
>>>>
>>>>> Here is the rpm I was using at the time:
>>>>>
http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm
>>>>>
>>>>> To see the list of the patches and what order to apply
them, see the
>>>>> spec file.
>>>> That still won''t tell us which patches you did apply.
>>>>
>>>>> Please make sure this performance issue is fixed for the
4.2 release.
>>>>> And I would be happy to test whatever files you send me.
>>>> The sort of report you''re doing isn''t that
helpful. What would
>>>> help is if you could narrow down which patch(es) it is that
>>>> make things so much better. Giving 4.1.3-rc a try might also
>>>> be worthwhile, albeit I would hope we don''t have a
regression
>>>> in 4.2.0-rc compared to 4.1.3-rc...
>>>>
>>>> Jan
>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xen.org
>>>> http://lists.xen.org/xen-devel
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
Peter Maloney
2012-Oct-03 20:05 UTC
Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow
I also tested: modprobe xen-acpi-processor as suggeted by pasik on freenode. This didn''t help (tested with xen 4.3-unstable) And then also with 4.3-unstable, I tested 64 bit *windows8* preview, *which seemed to run fast. * So I guess this issue is specific to windows xp, or 32 bit domu in a 64 bit machine. On 10/03/2012 07:19 PM, Peter Maloney wrote:> I ran some new tests... 4.1.2 with different patches, and > 4.3-unstable.Some details are below. > > At some point in the future, I will try some builds between 4.1 and 4.2 > (but at the moment am not sure how with mercurial or what options I have). > > > > 4.1.2 > > short version: dom0 works fine; domu ran only in a few builds and works fine > > long version: > I tested 4.1.2 again, with a few selections of patches (first n patches > where n was; 0, 23, 46, 93, 186, 279; there are 373 patches in total). > All of them ran fast in dom0, unlike when I first started this mailing > list thread, and the builds that would run my windows domu ran it fast. > So probably there was something strange with the kernel I had before, > which was probably 3.4.4; now I''m using something like 3.5.x (cmason''s > linux-btrfs repository, for-linus branch) > > > > 4.3-unstable > > short version: dom0 works fine; domu always runs terribly slow (which > leads me to wanting to test what changed between 4.1 and 4.2) > > > long version: > I pulled the latest source, built it, and dom0 is fast just like with > 4.2, but windows hvm domu is still terribly slow, and (with 7 vcpus), it > consumes between 500-650% while booting and a few minutes afterwards. > With 4 CPUs, I would expect between 350-550% from observations with 4.2 > but didn''t test other cpu counts with 4.3. (another side note, with > 4.1.3 which is normally fast, it will run slow like 4.2 and 4.3 if I set > cpus="2,4,6,8" instead of cpus=4) > > xentop - 11:32:44 Xen 4.3-unstable > 2 domains: 2 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown > Mem: 16757972k total, 16739788k used, 18184k free CPUs: 8 @ 4499MHz > NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) > MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR > VBD_RSECT VBD_WSECT SSID > Domain-0 -----r 784 27.2 12314624 73.5 12582912 > 75.1 8 0 0 0 0 0 0 > 0 0 0 0 > windowsxp2 -----r 3244 637.1 4197220 25.0 4198400 > 25.1 7 1 344 56 2 0 14381 6054 > 651283 122280 0 > > And then after idling for 10 minutes, it is under 200% > > xentop - 11:35:29 Xen 4.3-unstable > 2 domains: 1 running, 1 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown > Mem: 16757972k total, 16739788k used, 18184k free CPUs: 8 @ 4499MHz > NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) > MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR > VBD_RSECT VBD_WSECT SSID > Domain-0 -----r 839 42.4 12314624 73.5 12582912 > 75.1 8 0 0 0 0 0 0 > 0 0 0 0 > windowsxp2 --b--- 3583 114.1 4197220 25.0 4198400 > 25.1 7 1 426 66 2 0 14408 7501 > 651853 180372 0 > > > And then when it is in use (just loading a youtube page), it is up high > again. > > xentop - 11:37:17 Xen 4.3-unstable > 2 domains: 2 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown > Mem: 16757972k total, 16739788k used, 18184k free CPUs: 8 @ 4499MHz > NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) > MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR > VBD_RSECT VBD_WSECT SSID > Domain-0 -----r 875 37.8 12314624 73.5 12582912 > 75.1 8 0 0 0 0 0 0 > 0 0 0 0 > windowsxp2 -----r 3945 529.7 4197220 25.0 4198400 > 25.1 7 1 4885 201 2 0 17096 8168 > 788573 198458 0 > > > And also if I shut down the vm while it is at 600% cpu, it takes > something like 10-15 minutes to shut down. > > and CPU temperature is only 45 degrees during the high cpu usage, and > 26-32 degrees when it''s 0%, so whatever CPU waste it''s doing is not > generating much heat. With 4.1.3, while a game is open, it reports 200% > cpu, and the temperature is around 50 degrees. I have a huge CPU cooler; > it''s overclocked; and normally it runs about 55-70 degrees using 8 cores > depending on the task. > > I tested with apic=0 and apic=1 (apic=1 will run windows very slow in > 4.1.x, so I have been using apic=0 normally) > > > > > > > > On 08/13/2012 10:59 PM, Peter Maloney wrote: >> I also tested 4.1.3, which is fast, and both USB and graphics >> passthrough work, but "xl create" gave this message the first time I >> started the vm (but not the second): >> >> libxl: error: libxl_pci.c:750:libxl_device_pci_reset The kernel doesn''t >> support reset from sysfs for PCI device 0000:00:12.0 >> >> >> 0000:00:12.0 is a USB device, which works in the VM. >> >> peter:/opt # lspci -v | grep 00:12.0 >> 00:12.0 USB Controller: ATI Technologies Inc SB7x0/SB8x0/SB9x0 USB OHCI0 >> Controller (prog-if 10 [OHCI]) >> >> >> On 08/13/2012 08:54 PM, Peter Maloney wrote: >>> So... did my 4.2-unstable test, using a fresh pull from yesterday; dom0 >>> is normal fast (unlike previous tests), and domU is ultra slow, but >>> actually boots, and graphics card passthrough works without any patches, >>> and so does the USB keyboard, but USB mouse passthrough doesn''t work. >>> >>> >>> On 08/07/2012 09:25 AM, Peter Maloney wrote: >>>>> That still won''t tell us which patches you did apply. >>>> I applied no patches and tested, and the result was slow. And then >>>> applied all patches, and it was fast. I didn''t try figuring out which >>>> one it was. >>>> >>>> >>>> So I guess I''ll try: >>>> - the latest unstable 4.2 >>>> - the 4.1.3-rc (Which includes the patch Malcolm suggested) >>>> - and my rpm source with half patches, 3/4 of them, etc. binary search >>>> style to see which patch(es) changed the performance. But this means I >>>> won''t be able to narrow it down to a single patch, but only the point in >>>> the long list where the most dramatic change happens, possibly depending >>>> on many previous patches. >>>> >>>> Thanks so far, guys. >>>> >>>> >>>> On 08/06/2012 12:31 PM, Jan Beulich wrote: >>>>>>>> On 06.08.12 at 12:12, Peter Maloney <peter.maloney@brockmann-consult.de> wrote: >>>>>> my AMD FX-8150 system with vanilla source code is super slow, both the >>>>>> dom0 and domUs. However, after I merge the upstream patches I found in >>>>>> the openSUSE rpm, it runs normally. >>>>> I''d be very surprised if you really just took the upstream patches, >>>>> and the result was better than 4.2-rc1. After all, what upstream >>>>> means is that they were taken from -unstable. >>>>> >>>>>> I tried 4.2-unstable and it was the same. There was no rc1 when I tested >>>>>> it about 1.5 weeks ago. And 4.2 has the same horrible performance, and >>>>>> obviously those patches won''t work any more since the 4.2 code looks >>>>>> completely reorganized, so I''m stuck with 4.1.2 >>>>> Obviously the upstream patches can''t be applied to something >>>>> that already has all those changes. Other patches, of which we >>>>> unfortunately have quite a few, would be a different story. >>>>> >>>>>> Here is the rpm I was using at the time: >>>>>> http://download.opensuse.org/update/12.1/src/xen-4.1.2_16-1.7.1.src.rpm >>>>>> >>>>>> To see the list of the patches and what order to apply them, see the >>>>>> spec file. >>>>> That still won''t tell us which patches you did apply. >>>>> >>>>>> Please make sure this performance issue is fixed for the 4.2 release. >>>>>> And I would be happy to test whatever files you send me. >>>>> The sort of report you''re doing isn''t that helpful. What would >>>>> help is if you could narrow down which patch(es) it is that >>>>> make things so much better. Giving 4.1.3-rc a try might also >>>>> be worthwhile, albeit I would hope we don''t have a regression >>>>> in 4.2.0-rc compared to 4.1.3-rc... >>>>> >>>>> Jan >>>>> >>>>> >>>>> _______________________________________________ >>>>> Xen-devel mailing list >>>>> Xen-devel@lists.xen.org >>>>> http://lists.xen.org/xen-devel >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xen.org >>> http://lists.xen.org/xen-devel >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Konrad Rzeszutek Wilk
2012-Oct-04 14:25 UTC
Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow
On Wed, Oct 03, 2012 at 10:05:24PM +0200, Peter Maloney wrote:> I also tested: > modprobe xen-acpi-processorYou didn''t say what kind of CPU you have. Nor if you compiled your kernel or if you used a distros'' kernel. One thing (just to eliminate this being a power management issue), is to do this on the Linux command line: xen-acpi-processor.off=1 It would also help if you provided the .config you are using. There are some options you should not have one (like XEN_DEBUGFS.. something).
Peter Maloney
2012-Oct-05 20:19 UTC
Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow
On 10/04/2012 04:25 PM, Konrad Rzeszutek Wilk wrote:> On Wed, Oct 03, 2012 at 10:05:24PM +0200, Peter Maloney wrote: >> I also tested: >> modprobe xen-acpi-processor > You didn''t say what kind of CPU you have. Nor if you compiled > your kernel or if you used a distros'' kernel. > > One thing (just to eliminate this being a power management issue), > is to do this on the Linux command line: > > xen-acpi-processor.off=1 > > It would also help if you provided the .config you are using. There > are some options you should not have one (like XEN_DEBUGFS.. something). >AMD FX-8150 990 FX chipset I compiled the for-linus branch of cmason''s linux-btrfs git repo ( git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git for-linus ) Here''s some hardware info: http://pastebin.com/XUZjmiVz Here''s the kernel config: http://pastebin.com/1GQbiFZE (only weird thing I set was CONFIG_NR_CPUS=16 for no particular reason; default was 512 or 256) 3d stuff works fine (other than no es1370 sound device support) and fast in windows8 preview also, without that 0fps ''warmup'' / ''hicup'' thing happening (apparently as new textures are loading), which I wrote about before in IRC but not in the mailing list. At some point I plan on also testing: - builds between 4.1 and xen 4.2 to find where the 32 bit windowsxp starts going slow - a 64 bit windows xp if I can find one... I don''t have one at the moment - a 32 bit windows 8 preview (I assume I can just download this like 64 bit win8 preview)
Peter Maloney
2012-Oct-06 08:36 UTC
Re: 4.1.2 very slow without upstream patches, but fast with them, also 4.2 very slow
On 10/05/2012 10:19 PM, Peter Maloney wrote:> On 10/04/2012 04:25 PM, Konrad Rzeszutek Wilk wrote: >> On Wed, Oct 03, 2012 at 10:05:24PM +0200, Peter Maloney wrote: >>> I also tested: >>> modprobe xen-acpi-processor >> You didn''t say what kind of CPU you have. Nor if you compiled >> your kernel or if you used a distros'' kernel. >> >> One thing (just to eliminate this being a power management issue), >> is to do this on the Linux command line: >> >> xen-acpi-processor.off=1 >> >> It would also help if you provided the .config you are using. There >> are some options you should not have one (like XEN_DEBUGFS.. something). >> > AMD FX-8150 > 990 FX chipset > I compiled the for-linus branch of cmason''s linux-btrfs git repo ( > git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git > for-linus ) > > Here''s some hardware info: http://pastebin.com/XUZjmiVz > Here''s the kernel config: http://pastebin.com/1GQbiFZE (only weird thing > I set was CONFIG_NR_CPUS=16 for no particular reason; default was 512 or > 256) > > > 3d stuff works fine (other than no es1370 sound device support) and fast > in windows8 preview also, without that 0fps ''warmup'' / ''hicup'' thing > happening (apparently as new textures are loading), which I wrote about > before in IRC but not in the mailing list. > > At some point I plan on also testing: > - builds between 4.1 and xen 4.2 to find where the 32 bit windowsxp > starts going slow > - a 64 bit windows xp if I can find one... I don''t have one at the moment > - a 32 bit windows 8 preview (I assume I can just download this like 64 > bit win8 preview)32 bit Windows 8 preview is fast... so then it''s only Windows xp that is slow so far. And es1370 sound also works in 32 bit win8.