Pkg xen devel - Sep 2015 - Bug#799122: xen-hypervisor-4.4-amd64: Networking of domUs stops working after a few minutes

Package: xen-hypervisor-4.4-amd64
Version: 4.4.1-9+deb8u1
Severity: important

A few minutes after starting a domU, network access is no longer possible from
and to it.

This does not always happen and is not easily reproducible, but seems to occur
in all newly started domUs from some point in time on. However, also restarting
the dom0 does not necessarily prevent the problem.

At the moment when the network in the domU completely stops working, there is
the error message
[2178752.854380] vif vif-33-0 vif33.0: Guest Rx stalled
visible in dmesg in the dom0.

It is sometimes possible to for example ping the domU for a longer time than
pinging any host from the domU. Also pings may still be possible for a few
minutes, while SSH sessions do no longer work.

We tested the current lenny kernel linux-image-3.16.0-4-amd64 as well as the
backport linux-image-4.1.0-0.bpo.1-amd64 on the dom0 as well as the domU. The
problem happens with newly created domUs via 'xen-create-image', as well
as with older domUs which have been migrated from a debian wheezy dom0. It
happens with the vif-route as well as the vif-bridge script in the domU
configuration.

When the network stops working, the ARP tables are no longer filled on dom0 and
domU, for example:

Address                  HWtype  HWaddress           Flags Mask            Iface
x.y.z.v                     (incomplete)                              vif33.0


-- System Information:
Debian Release: 8.2
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.1.0-0.bpo.1-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

xen-hypervisor-4.4-amd64 depends on no packages.

Versions of packages xen-hypervisor-4.4-amd64 recommends:
ii  xen-utils-4.4  4.4.1-9+deb8u1

xen-hypervisor-4.4-amd64 suggests no packages.

-- no debconf information

On Wed, 2015-09-16 at 01:52 +0200, Arne Klein wrote:
> 
[...]
> At the moment when the network in the domU completely stops working,
> there is the error message
> [2178752.854380] vif vif-33-0 vif33.0: Guest Rx stalled
> visible in dmesg in the dom0.
This will therefore be a kernel issue not a hypervisor one. It sounds
like a backend one given the issue is with both new and existing Wheezy
guests.
> We tested the current lenny kernel linux-image-3.16.0-4-amd64 as well
> as the backport linux-image-4.1.0-0.bpo.1-amd64 on the dom0 as well
> as the domU.
(I suppose you meant s/lenny/jessie/ ;-)

These are kernel ABI versions, the package release versions are things
like 3.16.7-ckt17-1 or 4.2.5-1~bpo8+1, which yu can either get from
dpkg or from /proc/version (at the end, before the date, I think).

If you can let me know the versions then I can more sensibly reassign
this to the kernel packages. It will also give some baselines to see
what if any fixes we do or don't have.
>  The problem happens with newly created domUs via
'xen-create-image',
> as well as with older domUs which have been migrated from a debian
> wheezy dom0. It happens with the vif-route as well as the vif-bridge
> script in the domU configuration.
> 
> When the network stops working, the ARP tables are no longer filled
> on dom0 and domU, for example:
> 
> Address??????????????????HWtype??HWaddress???????????Flags
> Mask????????????Iface
> x.y.z.v?????????????????????(incomplete)?????????????????????????????
> ?vif33.0
> 
> 
> -- System Information:
> Debian Release: 8.2
> ? APT prefers stable-updates
> ? APT policy: (500, 'stable-updates'), (500, 'stable')
> Architecture: amd64 (x86_64)
> 
> Kernel: Linux 4.1.0-0.bpo.1-amd64 (SMP w/8 CPU cores)
> Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
> Shell: /bin/sh linked to /bin/dash
> Init: systemd (via /run/systemd/system)
> 
> xen-hypervisor-4.4-amd64 depends on no packages.
> 
> Versions of packages xen-hypervisor-4.4-amd64 recommends:
> ii??xen-utils-4.4??4.4.1-9+deb8u1
> 
> xen-hypervisor-4.4-amd64 suggests no packages.
> 
> -- no debconf information
> 
> _______________________________________________
> Pkg-xen-devel mailing list
> Pkg-xen-devel at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-xen-devel
>

>> We tested the current lenny kernel linux-image-3.16.0-4-amd64 as well
>> as the backport linux-image-4.1.0-0.bpo.1-amd64 on the dom0 as well
>> as the domU.
>
> (I suppose you meant s/lenny/jessie/ ;-)
Oops, yes :)
> These are kernel ABI versions, the package release versions are things
> like 3.16.7-ckt17-1 or 4.2.5-1~bpo8+1, which yu can either get from
> dpkg or from /proc/version (at the end, before the date, I think).
>
> If you can let me know the versions then I can more sensibly reassign
> this to the kernel packages. It will also give some baselines to see
> what if any fixes we do or don't have.
Thank you. The tested versions on dom0 and domU in which the problem 
occurs are:
4.1.3-1~bpo8+1
3.16.7-ckt11-1+deb8u5

I posted the problem also to the xen-user mailing list, but did not get 
a reply there.

And I have to update one of the observations: It seems that not all 
domUs started after the problem occurs for the first time are broken. 
After several restarts (without any changes) one of the domUs started 
working again.

Processing control commands:
> reassign -1 src:linux 3.16.7-ckt11-1+deb8u5
Bug #799122 [xen-hypervisor-4.4-amd64] xen-hypervisor-4.4-amd64: Networking of
domUs stops working after a few minutes
Bug reassigned from package 'xen-hypervisor-4.4-amd64' to
'src:linux'.
No longer marked as found in versions xen/4.4.1-9+deb8u1.
Ignoring request to alter fixed versions of bug #799122 to the same values
previously set
Bug #799122 [src:linux] xen-hypervisor-4.4-amd64: Networking of domUs stops
working after a few minutes
The source 'linux' and version '3.16.7-ckt11-1+deb8u5' do not
appear to match any binary packages
Marked as found in versions linux/3.16.7-ckt11-1+deb8u5.
> found -1?4.1.3-1~bpo8+1
Bug #799122 [src:linux] xen-hypervisor-4.4-amd64: Networking of domUs stops
working after a few minutes
Marked as found in versions linux/4.1.3-1~bpo8+1.

-- 
799122: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=799122
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems