Xen users - Jul 2012 - xm top shows high CPU load on domU when passed PCI NICs after hypervisor update

I am running a 64 bit dom0 with a stock Debian Squeeze xen hypervisor
and kernel.  On top of this I am running a PV 32 bit domU that is
passed two PCI network cards.  This set-up had been working well for
over a year, however there was a recent hypervisor update so I
rebooted the dom0.  Unfortunately now the domU that is being passed
the two NICs is no longer able to create a PPPoe connection to my ISP
and xm top show''s it''s CPU use consistently at 100%. 
Suggestions on
what I can do to trouble shoot this would be appreciated.  Here is
what I have tried to date:

- Revert the hypervisor update (packages
libxenstore3.0_4.0.1-4_amd64.deb
xen-hypervisor-4.0-amd64_4.0.1-4_amd64.deb
xenstore-utils_4.0.1-4_amd64.deb xen-utils-4.0_4.0.1-4_amd64.deb
xen-utils-common_4.0.0-1_all.deb) and reboot the dom0, but this
doesn''t seem to make any difference, so perhaps the timing was a
coincidence or I''ve missed reverting a package.
- Booting the domU without passing PCI cards to it, it boots normally
and shows normal CPU load via xm top.  This suggests the problem is
with PCI pass through.
- Passing only 1 PCI card at a time to the domU to see if the issue
was card specific, however passing either card shows the same CPU load
issue so the problem is not PCI card specific
- Passing one of the PCI cards to another domU, which then shows the
exact same CPU use issue as the original domU, so the issue isn''t domU
specific

Other curious observations:
- top run on the domU does not show the high CPU load, however
restarting networking on the domU takes much longer than normal and as
mentioned earlier, it is unable to create a PPPoe connection over the
passed network cards.

Other than troubleshooting, I haven''t made any configuration changes
to either the dom0 or domU.

Searching through the mailing list and google in general hasn''t turned
up reports of a simliar issue.

Any suggestions would be greatly appreciated.

- Ben

On Thu, 2012-07-26 at 21:26 +0100, Beanjammin H wrote:
> I am running a 64 bit dom0 with a stock Debian Squeeze xen hypervisor
> and kernel.  On top of this I am running a PV 32 bit domU that is
> passed two PCI network cards.  This set-up had been working well for
> over a year, however there was a recent hypervisor update so I
> rebooted the dom0.  Unfortunately now the domU that is being passed
> the two NICs is no longer able to create a PPPoe connection to my ISP
> and xm top show''s it''s CPU use consistently at 100%. 
Suggestions on
> what I can do to trouble shoot this would be appreciated.  Here is
> what I have tried to date:
> 
> - Revert the hypervisor update (packages
> libxenstore3.0_4.0.1-4_amd64.deb
> xen-hypervisor-4.0-amd64_4.0.1-4_amd64.deb
> xenstore-utils_4.0.1-4_amd64.deb xen-utils-4.0_4.0.1-4_amd64.deb
> xen-utils-common_4.0.0-1_all.deb) and reboot the dom0, but this
> doesn''t seem to make any difference, so perhaps the timing was a
> coincidence or I''ve missed reverting a package.
You might also need to try reverting any dom0 kernel updates which
happened in the interim as well?

Likewise domU.
> Searching through the mailing list and google in general hasn''t
turned
> up reports of a simliar issue.
There was one performance issue relating to swiotlb and bounce buffering
not so long ago. I''m hoping that Konrad (CCd) will remember the
details.

What are the exact dom0 and domU kernel versions?
> Any suggestions would be greatly appreciated.
> 
> - Ben
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xen.org
> http://lists.xen.org/xen-users

On Fri, Jul 27, 2012 at 09:39:23AM +0100, Ian Campbell
wrote:
> On Thu, 2012-07-26 at 21:26 +0100, Beanjammin H wrote:
> > I am running a 64 bit dom0 with a stock Debian Squeeze xen hypervisor
> > and kernel.  On top of this I am running a PV 32 bit domU that is
> > passed two PCI network cards.  This set-up had been working well for
> > over a year, however there was a recent hypervisor update so I
> > rebooted the dom0.  Unfortunately now the domU that is being passed
> > the two NICs is no longer able to create a PPPoe connection to my ISP
> > and xm top show''s it''s CPU use consistently at 100%.
Suggestions on
> > what I can do to trouble shoot this would be appreciated.  Here is
> > what I have tried to date:
> > 
> > - Revert the hypervisor update (packages
> > libxenstore3.0_4.0.1-4_amd64.deb
> > xen-hypervisor-4.0-amd64_4.0.1-4_amd64.deb
> > xenstore-utils_4.0.1-4_amd64.deb xen-utils-4.0_4.0.1-4_amd64.deb
> > xen-utils-common_4.0.0-1_all.deb) and reboot the dom0, but this
> > doesn''t seem to make any difference, so perhaps the timing
was a
> > coincidence or I''ve missed reverting a package.
> 
> You might also need to try reverting any dom0 kernel updates which
> happened in the interim as well?
> 
> Likewise domU.
> 
> > Searching through the mailing list and google in general
hasn''t turned
> > up reports of a simliar issue.
> 
> There was one performance issue relating to swiotlb and bounce buffering
> not so long ago. I''m hoping that Konrad (CCd) will remember the
details.
It was an optimization:

commit 6810df88dcfc22de267caf23eb072ffb97b3c411
Author: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date:   Thu Aug 25 16:13:54 2011 -0400

    xen-swiotlb: When doing coherent alloc/dealloc check before swizzling the
MFNs.
    
> 
> What are the exact dom0 and domU kernel versions?
> 
> > Any suggestions would be greatly appreciated.
> > 
> > - Ben
> > 
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@lists.xen.org
> > http://lists.xen.org/xen-users
>