Xen devel - Jun 2011 - AW: Load increase after memory upgrade?

Hi,

the higher load is neither bound to memory increase nor Xen version,
it''s the kernel.
Changed at the same time from a Xenified 2.6.34.7 kernel to pvops
jeremy''s git
stable-2.6.32.x

Konrad, are you interested? You can see from attached graphic the much higer
load
mid 21 to 24. riker (yellow) is using the cards constantly for scanning
transponders,
two seem to make use of DMA for that reason...

Carsten.


----- Originalnachricht -----
Von: Carsten Schiers <carsten@schiers.de>
Gesendet: Don, 23.6.2011 16:22
An: xen-devel <xen-devel@lists.xensource.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Betreff: Load increase after memory upgrade?

Hi,

after my memory upgrade 4GB -> 8GB, I noticed a much higher load in the Video
DomU,
related to some PCI cards that do DMA. I could check that it''s related
to two of the
three cards that use the same hardware architecture (budget_av and budget_ci)
whereas
the b2c2_flexcop doesn''t show shuch a behaviour.

As I changed at the same time from Xen 4.1.0 to 4.1.1, I wondered whether this
is

  a) normal or pathologic
  b) related to Xen or the memory upgrade

Any thoughts? Can be seen as the yellow domain in my munin graphs.

Carsten.




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Jun 27, 2011 at 01:29:24PM +0200, Carsten Schiers
wrote:
> Hi,
> 
> the higher load is neither bound to memory increase nor Xen version,
it''s the kernel.
> Changed at the same time from a Xenified 2.6.34.7 kernel to pvops
jeremy''s git
> stable-2.6.32.x
> 
> Konrad, are you interested? You can see from attached graphic the much
higer load
> mid 21 to 24. riker (yellow) is using the cards constantly for scanning
transponders,
> two seem to make use of DMA for that reason...
Um, so 2.6.34 to 2.6.32 causes this spike in usage? What is the SWIOTLB
(PCI-DMA32)
usage between these two kernels and are the drivers (budget_...) the same
version?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hello Konrad,

Monday, June 27, 2011, 3:46:43 PM, you wrote:
> On Mon, Jun 27, 2011 at 01:29:24PM +0200, Carsten Schiers wrote:
>> Hi,
>> 
>> the higher load is neither bound to memory increase nor Xen version,
it''s the kernel.
>> Changed at the same time from a Xenified 2.6.34.7 kernel to pvops
jeremy''s git
>> stable-2.6.32.x
>> 
>> Konrad, are you interested? You can see from attached graphic the much
higer load
>> mid 21 to 24. riker (yellow) is using the cards constantly for scanning
transponders,
>> two seem to make use of DMA for that reason...
> Um, so 2.6.34 to 2.6.32 causes this spike in usage? What is the SWIOTLB
(PCI-DMA32)
> usage between these two kernels and are the drivers (budget_...) the same
version?
is it actually the drivers that are consuming more cpu ?
2.6.32.x series still seem to show higher cpu utilazation and more wake-ups due
to some console issue resulting in high events/0 in dom0.
That seems to be over in the new linux 3.0 kernels

--
Sander


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>Um, so 2.6.34 to 2.6.32 causes this spike in usage? 
Definitely yes. 
> What is the SWIOTLB (PCI-DMA32) usage between these two kernels 
I don''t understand. What do you mean? The only difference is that I
call
up the swiotlb differently:

On pvops: iommu=soft
On rebased: swiotlb=32,force

BTW: changing to 64,force doesn''t make a difference.

I attached a normal boot log of both kernels.
> and are the drivers (budget_...) the same version?
No, but a quick diff between the two source trees offer only minor 
changes that from the first look
do not seem to be related to DMA or whatever.

Carsten.





_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>That seems to be over in the new linux 3.0 kernels
If you recomend a kernel, I can try. Is jeremy''s xen/next-3.0 capable
to
to pci-passthrough?

Carsten.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Monday, June 27, 2011, 10:00:43 PM, you wrote:
>>That seems to be over in the new linux 3.0 kernels
> If you recomend a kernel, I can try. Is jeremy''s xen/next-3.0
capable to
> to pci-passthrough?
I''m running konrad''s master
tree(git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git) with a
additional patch that''s the last one in his
"devel/xen-pciback-0.6.1" tree (xen/pciback: revert the name of the
PCI driver to ...).
The patch is necessary, otherwise the xen toolstack doesn''t see the
devices being owned by pciback.

> Carsten.



-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>I''m running konrad''s master 
tree(git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git) with 
a additional patch that''s the last one in his
"devel/xen-pciback-0.6.1"
tree >(xen/pciback: revert the name of the PCI driver to
...).
>The patch is necessary, otherwise the xen toolstack doesn''t see the
devices being owned by pciback.

Checking tomorrow. It seems the patch you mention is now already in (2 
hours ago) ;o).

Carsten.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>I''m running konrad''s master 
tree(git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git) with 
a additional patch that''s the last one in his
"devel/xen-pciback-0.6.1"
tree >(xen/pciback: revert the name of the PCI driver to
...).
>The patch is necessary, otherwise the xen toolstack doesn''t see the
devices being owned by pciback.

Forget what I just mailed. It''s a head. I am not very used to all these
git thingies...

Is there an easy way to describe the steps to do after

  git clone git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git

?

Carsten.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Jun 27, 2011 at 09:56:38PM +0200, Carsten Schiers
wrote:
> >Um, so 2.6.34 to 2.6.32 causes this spike in usage? 
> 
> Definitely yes. 
What happens if you remove the extra 4GB of memory? You mentioned you did
two things: a) more memory and b) change kernel versions.

Need to narrow down which one is at fault here.
> 
> > What is the SWIOTLB (PCI-DMA32) usage between these two kernels 
> 
> I don''t understand. What do you mean? The only difference is that
I call
> up the swiotlb differently:
> 
> On pvops: iommu=soft
so 64MB here
> On rebased: swiotlb=32,force
32MB here.

Which is OK. I was somehow thinking you might have a small amount of
memory in the pvops case - but you actually have a pretty big buffer there.
> 
> BTW: changing to 64,force doesn''t make a difference.
<nods> So DMA might not be an issue here..
> 
> I attached a normal boot log of both kernels.
> 
> > and are the drivers (budget_...) the same version?
> 
> No, but a quick diff between the two source trees offer only minor 
> changes that from the first look
> do not seem to be related to DMA or whatever.
> 
> Carsten.
> 
> 

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Jun 27, 2011 at 10:00:43PM +0200, Carsten Schiers
wrote:
> >That seems to be over in the new linux 3.0 kernels
> 
> If you recomend a kernel, I can try. Is jeremy''s xen/next-3.0
capable to
> to pci-passthrough?
Any 2.6.37 or later kernel has the PCI passthrough frontend driver in
it.
> 
> Carsten.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Jun 27, 2011 at 10:46:15PM +0200, Carsten Schiers
wrote:
> >I''m running konrad''s master 
> tree(git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git) with 
> a additional patch that''s the last one in his
"devel/xen-pciback-0.6.1"
> tree >(xen/pciback: revert the name of the PCI driver to ...).
> >The patch is necessary, otherwise the xen toolstack doesn''t
see the
> devices being owned by pciback.
> 
> Forget what I just mailed. It''s a head. I am not very used to all
these
> git thingies...
> 
> Is there an easy way to describe the steps to do after
> 
>   git clone git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git

Um, well, lets take one thing at a time.

a). What is the version of Dom0? Are you swapping that out? If so, and you
want the latests then the next step is to do:

cd xen
git merge origin/devel/xen-pciback-0.6.1
make

b). If you just want to use the later version of a kernel for DomU, you
don''t
need the xen pciback. You need the xen pcifront - which has some bug-fixes in
2.6.39.
So you can:

cd xen
git checkout v2.6.39
make

c). If you want to see if the fault here lies in the bounce buffer being used
more
often in the DomU b/c you have 8GB of memory now and you end up using more pages
past 4GB (in DomU), I can cook up a patch to figure this out. But an easier way
is
to just do (on the Xen hypervisor line): mem=4G and that will make think you
only have
4GB of physical RAM.  If the load comes back to the normal "amount"
then the likely
culprit is that and we can think on how to fix this.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>b). If you just want to use the later version of a kernel for DomU, you 
don''t
>need the xen pciback. You need the xen pcifront - which has some 
bug-fixes in 2.6.39.

I started with this one. First observation: seems to generate high load 
(~20% in xentop, as opposed
to ~10% with the Xenified kernel).

Xen 4.1.1 with stable/2.6.39.x from Konrad xen.git.

Also a WARNING came up during boot, but everything works. But I thought 
you would eventually be able to 
read something from it:

[    2.850254] WARNING: at kernel/irq/handle.c:130 
handle_irq_event_percpu+0xa0/0x15f()
[    2.850274] irq 28 handler interrupt_hw+0x0/0x1b7 [saa7146] enabled 
interrupts
[    2.850288] Modules linked in: budget_ci(+) b2c2_flexcop(+) 
ir_rc5_decoder budget_core dvb_core ir_nec_decoder snd_pcm cx24123 
saa7146 snd_timer cx24113 snd ttpci_eeprom s5h1420 soundcore rc_core 
snd_page_alloc i2c_core pcspkr ext3 jbd mbcache xen_netfront 
xen_blkfront
[    2.850374] Pid: 141, comm: udevd Not tainted 2.6.39.2-xen-686 #4
[    2.850384] Call Trace:
[    2.850398]  [<c10361e8>] ? warn_slowpath_common+0x6a/0x7b
[    2.850411]  [<c107536b>] ? handle_irq_event_percpu+0xa0/0x15f
[    2.850424]  [<c103625f>] ? warn_slowpath_fmt+0x28/0x2c
[    2.850437]  [<c107536b>] ? handle_irq_event_percpu+0xa0/0x15f
[    2.850451]  [<d51cfe32>] ? SAA7146_IER_DISABLE+0x36/0x36 [saa7146]
[    2.850465]  [<c107544b>] ? handle_irq_event+0x21/0x37
[    2.850478]  [<c1076d36>] ? handle_fasteoi_irq+0x66/0x7e
[    2.850492]  [<c11a8515>] ? __xen_evtchn_do_upcall+0x129/0x1b0
[    2.850506]  [<c11a913b>] ? xen_evtchn_do_upcall+0x18/0x26
[    2.850519]  [<c12aea97>] ? xen_do_upcall+0x7/0xc
[    2.850532]  [<c114b407>] ? strcmp+0x8/0x19
[    2.850545]  [<c1104287>] ? sysfs_dentry_revalidate+0x38/0x88
[    2.850559]  [<c10ca1d1>] ? walk_component+0x18c/0x2b4
[    2.850571]  [<c1103437>] ? sysfs_permission+0x49/0x4f
[    2.850583]  [<c10ca96a>] ? link_path_walk+0x14f/0x38a
[    2.850597]  [<c1113e66>] ? security_file_alloc+0xc/0xd
[    2.850610]  [<c10cb9db>] ? path_openat+0x7f/0x27d
[    2.850623]  [<c10cbc7d>] ? do_filp_open+0x21/0x60
[    2.850635]  [<c10bdc68>] ? kmem_cache_alloc+0x21/0xd9
[    2.850647]  [<c10c921e>] ? getname_flags+0x20/0xbd
[    2.850660]  [<c10c04f4>] ? do_sys_open+0x58/0xd1
[    2.850672]  [<c10c05b1>] ? sys_open+0x1e/0x23
[    2.850684]  [<c12a901c>] ? syscall_call+0x7/0xb
[    2.850694] ---[ end trace 5b96651124c9fbda ]---




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>a). What is the version of Dom0? Are you swapping that out? If so, and 
you
>want the latests then the next step is to do:
Dom0 is jeremy''s git xen/stable-2.6.32.x in 64 Bit. DomUs are 32 Bit. 
Shall I 
experimently change that to what you recommended?

>c). If you want to see if the fault here lies in the bounce buffer 
being used more
>often in the DomU b/c you have 8GB of memory now and you end up using 
more pages
>past 4GB (in DomU), I can cook up a patch to figure this out. But an 
easier way is
>to just do (on the Xen hypervisor line): mem=4G and that will make 
think you only have
>4GB of physical RAM.  If the load comes back to the normal
"amount"
then the likely
>culprit is that and we can think on how to fix this.
So I will reboot with the a.m. parameter, but that''s not possible right
now. I 
will keep you informed. Eventually I can combine that with the test a) 
on Thursday.

Carsten.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>is it actually the drivers that are consuming more cpu ?
>2.6.32.x series still seem to show higher cpu utilazation and more 
wake-ups due to some console issue resulting in high events/0 in
dom0.
>That seems to be over in the new linux 3.0 kernels
It seems that with the lower cpu utilization, main reasons to wake up 
according to powertop are:

  1) timer (~50%)
  2) saa7146(0)
  3) saa7146(1)

Whereas with higher CPU utilization its:

  1) flexcop-pci
  2) saa7146(0)
  3) saa7146(1)
  4) timer (<10%)

If that helps. Funny: taking out the flexcop card will not change load, 
but taking out the saa7146 cards will lower load on the other kernel 
also.

Carsten.  


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Jun 28, 2011 at 10:22:34PM +0200, Carsten Schiers
wrote:
> >a). What is the version of Dom0? Are you swapping that out? If so, and 
> you
> >want the latests then the next step is to do:
> 
> Dom0 is jeremy''s git xen/stable-2.6.32.x in 64 Bit. DomUs are 32
Bit.
> Shall I 
> experimently change that to what you recommended?
Lets first do the c) experiment as that will likely explain your load average
increase.
> 
> 
> >c). If you want to see if the fault here lies in the bounce buffer 
> being used more
> >often in the DomU b/c you have 8GB of memory now and you end up using 
> more pages
> >past 4GB (in DomU), I can cook up a patch to figure this out. But an 
> easier way is
> >to just do (on the Xen hypervisor line): mem=4G and that will make 
> think you only have
> >4GB of physical RAM.  If the load comes back to the normal
"amount"
> then the likely
> >culprit is that and we can think on how to fix this.
> 
> So I will reboot with the a.m. parameter, but that''s not possible
right
> now. I 
> will keep you informed. Eventually I can combine that with the test a) 
> on Thursday.
> 
> Carsten.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Lets first do the c) experiment as that will likely explain your load 
average increase.
...
> >c). If you want to see if the fault here lies in the bounce buffer 
> being used more
> >often in the DomU b/c you have 8GB of memory now and you end up using 
> more pages
> >past 4GB (in DomU), I can cook up a patch to figure this out. But an 
> easier way is
> >to just do (on the Xen hypervisor line): mem=4G and that will make 
> think you only have
> >4GB of physical RAM.  If the load comes back to the normal
"amount"
> then the likely
> >culprit is that and we can think on how to fix this.
You are on the right track. Load was going down to "normal" 10% when 
reducing
Xen to 4GB by the parameter. Load seems to be still a little, little bit 
lower
with Xenified Kernel (8-9%), but this is drastically lower than the 20% 
we had
before.

What to do next, Konrad?

Carsten.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Jun 29, 2011 at 11:17:53PM +0200, Carsten Schiers
wrote:
> > Lets first do the c) experiment as that will likely explain your load 
> average increase.
> ...
> > >c). If you want to see if the fault here lies in the bounce buffer
> > being used more
> > >often in the DomU b/c you have 8GB of memory now and you end up
using
> 
> > more pages
> > >past 4GB (in DomU), I can cook up a patch to figure this out. But
an
> > easier way is
> > >to just do (on the Xen hypervisor line): mem=4G and that will make
> > think you only have
> > >4GB of physical RAM.  If the load comes back to the normal
"amount"
> > then the likely
> > >culprit is that and we can think on how to fix this.
> 
> You are on the right track. Load was going down to "normal" 10%
when
> reducing
> Xen to 4GB by the parameter. Load seems to be still a little, little bit 
> lower
> with Xenified Kernel (8-9%), but this is drastically lower than the 20% 
> we had
> before.
> 
> What to do next, Konrad?
Yikes. I was hoping that was not the problem but it seems so. The issue is
here (I think) is that the PCI device you have passed in, are 32-bit.
Meaning they can only address memory up to 4GB. You should see if this
is the case - if the device is a PCI card, it definitly can only do 32-bit.

If it is PCIe, then it might be that the driver has set the DMA mask
to be only up to 32-bit.

Back to the explanation ..So if your guest (or the guest application) is
entirely above the 4GB region any of DMA operations have to be
"bounced"
(or copied) to a region below 4GB. When the DMA operation is complete the
result is copied from below the 4GB back to the region above 8Gb.

Some drivers use the ''dma_alloc_coherent'' which ends up
allocating a
buffer that is below 4GB and use that all the time as to avoid the
bouncing. Mostly the IDE, SCSI and NICs do it.

One way that immediately comes in my mind to "fix" this is to force
the guest to be allocated from memory below 4GB. But not sure what
fancy parameters need to be turned on for this.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel